Possible libsyncml MoreData bug?

[jasta]

I am trying to investigate a MoreData bug in libsyncml 0.4.5 that I
can't immediately determine to have been fixed in the main repository or
in 0.4.6, however there are comments in the diff between 0.4.5 and 0.4.6
that lead me to suspect this is either a known bug or one needing more
research currently.

Specifically, it looks like any time <MoreData> is sent by libsyncml
(and acknowledged by the receiver by way of a 214 status), the following
SyncML message starts the item from the beginning, so a compliant SyncML
client would append the item's contents to some part of the item from
the previous message.  Also, it looks like there is the possibility that
at some point during a session with a lot of packages ending with
MoreData, the server will incorrectly abort the session stating "Command
is too large".  This seems to be related to the use of Alert 222
(NEXT_MESSAGE), which my SyncML library sends at least once due to an
empty package sent from libsyncml.

Does anyone have any insight into this bug?  I am willing to investigate
and fix myself, but I'm jut not sure what is happening yet.  It seems
like maybe this feature just isn't well tested?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Michael Bell]

Josh Guilfoyle schrieb:

> I am trying to investigate a MoreData bug in libsyncml 0.4.5 that I
> can't immediately determine to have been fixed in the main repository or
> in 0.4.6, however there are comments in the diff between 0.4.5 and 0.4.6
> that lead me to suspect this is either a known bug or one needing more
> research currently.

I fixed one MoreData probelm in revision 383. libsyncml 0.4.6 is from
revision 342. I don't know if this fixes all problems. I only fixed some
kind of a state management problem. Which comments do you mean?

> Specifically, it looks like any time <MoreData> is sent by libsyncml
> (and acknowledged by the receiver by way of a 214 status), the following
> SyncML message starts the item from the beginning, so a compliant SyncML
> client would append the item's contents to some part of the item from
> the previous message.

Ticket #1.

> Also, it looks like there is the possibility that
> at some point during a session with a lot of packages ending with
> MoreData, the server will incorrectly abort the session stating "Command
> is too large".  This seems to be related to the use of Alert 222
> (NEXT_MESSAGE), which my SyncML library sends at least once due to an
> empty package sent from libsyncml.

Ticket #2.

> Does anyone have any insight into this bug?  I am willing to investigate
> and fix myself, but I'm jut not sure what is happening yet.  It seems
> like maybe this feature just isn't well tested?

I use photos in my contacts and so I test this stuff every time I
synchronize my mobile :)

Nevertheless I'm really interested to fix such bugs. Can you please open
two bugs on http://libsyncml.opensync.org/ (you must register and login
before you can do this)?

BTW actually the ticket mailing list does not work for this tracker but
a ticket is persistent and I cannot ignore it silently ...

Best regards

Michael
--
_______________________________________________________________

Michael Bell                    Humboldt-Universitaet zu Berlin

Tel.: [phone]       ZE Computer- und Medienservice
Fax:  [phone]       Unter den Linden 6
michael.bell@...   D-10099 Berlin
_______________________________________________________________

X.509 CA Certificates / Wurzelzertifikate

http://ra.pki.hu-berlin.de


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Michael Bell]

Michael Bell schrieb:
> Josh Guilfoyle schrieb:

>> Specifically, it looks like any time <MoreData> is sent by libsyncml
>> (and acknowledged by the receiver by way of a 214 status), the
>> following SyncML message starts the item from the beginning, so a
>> compliant SyncML client would append the item's contents to some part
>> of the item from the previous message.
>
> Ticket #1.

The code to handle too large items is in sml_session.c. The function is
called _smlSessionStartCommandInternal. Another suspect is the function
_smlSessionFragmentSend and all of its callers because it handles the
splitting of a command.

Best regards

Michael
--
_______________________________________________________________

Michael Bell                    Humboldt-Universitaet zu Berlin

Tel.: [phone]       ZE Computer- und Medienservice
Fax:  [phone]       Unter den Linden 6
michael.bell@...   D-10099 Berlin
_______________________________________________________________

X.509 CA Certificates / Wurzelzertifikate

http://ra.pki.hu-berlin.de


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Michael Bell]

Michael Bell schrieb:
> Josh Guilfoyle schrieb:

>> Also, it looks like there is the possibility that at some point during
>> a session with a lot of packages ending with MoreData, the server will
>> incorrectly abort the session stating "Command is too large".  This
>> seems to be related to the use of Alert 222 (NEXT_MESSAGE), which my
>> SyncML library sends at least once due to an empty package sent from
>> libsyncml.
>
> Ticket #2.

Which peer acts as the server?

Best regards

Michael
--
_______________________________________________________________

Michael Bell                    Humboldt-Universitaet zu Berlin

Tel.: [phone]       ZE Computer- und Medienservice
Fax:  [phone]       Unter den Linden 6
michael.bell@...   D-10099 Berlin
_______________________________________________________________

X.509 CA Certificates / Wurzelzertifikate

http://ra.pki.hu-berlin.de


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Michael Bell]

Michael Bell schrieb:

> Michael Bell schrieb:
>> Josh Guilfoyle schrieb:
>
>>> Specifically, it looks like any time <MoreData> is sent by libsyncml
>>> (and acknowledged by the receiver by way of a 214 status), the
>>> following SyncML message starts the item from the beginning, so a
>>> compliant SyncML client would append the item's contents to some part
>>> of the item from the previous message.
>>
>> Ticket #1.
>
> The code to handle too large items is in sml_session.c. The function is
> called _smlSessionStartCommandInternal. Another suspect is the function
> _smlSessionFragmentSend and all of its callers because it handles the
> splitting of a command.

I discovered two things in _smlSessionFragmentSend until now:

1. Why does a parameter complete_size exists if the size of the item is
determined dynamically in the function?

2. If a new command is created to send the beginning of a splitted
command then there is the following line:

if (start == 0)
     frag_cmd->size = complete_size;

This means that the size of the new command is set to the original (too
large) size.

Perhaps I need more time to understand the code but these are the first
things which I discovered. Perhaps somebody knows this part of the code
a little bit better than I.

Best regards

Michael

P.S. do you have a wireshark trace of such a failing conversation?
--
_______________________________________________________________

Michael Bell                    Humboldt-Universitaet zu Berlin

Tel.: [phone]       ZE Computer- und Medienservice
Fax:  [phone]       Unter den Linden 6
michael.bell@...   D-10099 Berlin
_______________________________________________________________

X.509 CA Certificates / Wurzelzertifikate

http://ra.pki.hu-berlin.de


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[jasta]

Michael,

Let me collect some more information and also try libsyncml SVN.  I have
the condition easily reproducible, so I will attempt to debug myself
first.  I'll let you know what I find and include Wireshark captures if
I am unable to figure it out easily.

BTW, my usage is from a client library I wrote in Java, and a server I
implemented in C using libsyncml.  I am simply using SyncML to implement
something entirely unrelated to contacts, calendar, etc, so perhaps my
unusual data set is demonstrating flaws that have previously gone
undetected.  Or perhaps it's my client library that is sending something
unusual causing libsyncml to behave this way.

Michael Bell wrote:

> Michael Bell schrieb:
>> Michael Bell schrieb:
>>> Josh Guilfoyle schrieb:
>>
>>>> Specifically, it looks like any time <MoreData> is sent by libsyncml
>>>> (and acknowledged by the receiver by way of a 214 status), the
>>>> following SyncML message starts the item from the beginning, so a
>>>> compliant SyncML client would append the item's contents to some
>>>> part of the item from the previous message.
>>>
>>> Ticket #1.
>>
>> The code to handle too large items is in sml_session.c. The function
>> is called _smlSessionStartCommandInternal. Another suspect is the
>> function _smlSessionFragmentSend and all of its callers because it
>> handles the splitting of a command.
>
> I discovered two things in _smlSessionFragmentSend until now:
>
> 1. Why does a parameter complete_size exists if the size of the item is
> determined dynamically in the function?
>
> 2. If a new command is created to send the beginning of a splitted
> command then there is the following line:
>
> if (start == 0)
>     frag_cmd->size = complete_size;
>
> This means that the size of the new command is set to the original (too
> large) size.
>
> Perhaps I need more time to understand the code but these are the first
> things which I discovered. Perhaps somebody knows this part of the code
> a little bit better than I.
>
> Best regards
>
> Michael
>
> P.S. do you have a wireshark trace of such a failing conversation?

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[jasta]

Sigh, I was using the wrong value for the "chunk accepted" status.

The SyncML protocol documentation available online[1] on page 17 states:

"On receipt of a data object with the <MoreData/> element, the recipient
MUST respond with a status response “214 – Chunked item accepted and
buffered” and, if there are no other commands to be sent, ask for the
next message using the Alert 222 mechanism defined in section 2.9."

However this is incorrect.  Elsewhere in the SyncML documentation it
states that 213 is the status, as libsyncml uses.


[1]
http://www.openmobilealliance.org/tech/affiliates/syncml/syncml_sync_protocol_v11_20020215.pdf

Josh Guilfoyle wrote:

> Michael,
>
> Let me collect some more information and also try libsyncml SVN.  I have
> the condition easily reproducible, so I will attempt to debug myself
> first.  I'll let you know what I find and include Wireshark captures if
> I am unable to figure it out easily.
>
> BTW, my usage is from a client library I wrote in Java, and a server I
> implemented in C using libsyncml.  I am simply using SyncML to implement
> something entirely unrelated to contacts, calendar, etc, so perhaps my
> unusual data set is demonstrating flaws that have previously gone
> undetected.  Or perhaps it's my client library that is sending something
> unusual causing libsyncml to behave this way.
>

-- snip --

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[jasta]

Having fixed that, my second problem still seems to be valid.  In
certain cases, the server will send an Add command with an incomplete
item and specify MoreData.  Then, on the next package, it will simply
send Status and Final, and libsyncml will emit SML_MANAGER_SESSION_END.
  My client then responds with Alert 222 (NEXT_MESSAGE), and libsyncml
resumes sending the incomplete item from the original message.

This may be my fault, however, as I think my usage of <Final> is
incorrect when sending status "213 - Chunk accepted".  See MsgID 5 from
the client.  Perhaps when I accept an incomplete item I should not mark
the package as Final?

I have attached a libpcap dump demonstrating this problem.  The Add is
in server MsgID 5, with the session-ending message following it as 6.

Josh Guilfoyle wrote:

> Sigh, I was using the wrong value for the "chunk accepted" status.
>
> The SyncML protocol documentation available online[1] on page 17 states:
>
> "On receipt of a data object with the <MoreData/> element, the recipient
> MUST respond with a status response “214 – Chunked item accepted and
> buffered” and, if there are no other commands to be sent, ask for the
> next message using the Alert 222 mechanism defined in section 2.9."
>
> However this is incorrect.  Elsewhere in the SyncML documentation it
> states that 213 is the status, as libsyncml uses.
>
>
> [1]
> http://www.openmobilealliance.org/tech/affiliates/syncml/syncml_sync_protocol_v11_20020215.pdf 
>
>
> Josh Guilfoyle wrote:
>> Michael,
>>
>> Let me collect some more information and also try libsyncml SVN.  I
>> have the condition easily reproducible, so I will attempt to debug
>> myself first.  I'll let you know what I find and include Wireshark
>> captures if I am unable to figure it out easily.
>>
>> BTW, my usage is from a client library I wrote in Java, and a server I
>> implemented in C using libsyncml.  I am simply using SyncML to
>> implement something entirely unrelated to contacts, calendar, etc, so
>> perhaps my unusual data set is demonstrating flaws that have
>> previously gone undetected.  Or perhaps it's my client library that is
>> sending something unusual causing libsyncml to behave this way.
>>
>
> -- snip --

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Michael Bell]

Josh Guilfoyle schrieb:

> Sigh, I was using the wrong value for the "chunk accepted" status.
>
> The SyncML protocol documentation available online[1] on page 17 states:
>
> "On receipt of a data object with the <MoreData/> element, the recipient
> MUST respond with a status response “214 – Chunked item accepted and
> buffered” and, if there are no other commands to be sent, ask for the
> next message using the Alert 222 mechanism defined in section 2.9."
>
> However this is incorrect.  Elsewhere in the SyncML documentation it
> states that 213 is the status, as libsyncml uses.

Ups, it is always a good idea to use the latest specification. The
problem is that the OMA takes over the management of the SyncML standard
after the release of v1.1.1. After this the OMA released a fixed version
1.1.2 but the OMA did not published an official errata list - in
contrast to the v1.0 releases :(

If you work on 1.1 support then I can only recommend you to use the
v1.1.2 spec but be warned the DataSyncProtocol link is damaged because
of a malformed file name. You must download the zip file with all four
files.

http://www.openmobilealliance.org/technical/release_program/ds_archive.aspx

Best regards

Michael

P.S. I sent an email to OMA technical support and asked them to fix the
link.
--
_______________________________________________________________

Michael Bell                    Humboldt-Universitaet zu Berlin

Tel.: [phone]       ZE Computer- und Medienservice
Fax:  [phone]       Unter den Linden 6
michael.bell@...   D-10099 Berlin
_______________________________________________________________

X.509 CA Certificates / Wurzelzertifikate

http://ra.pki.hu-berlin.de



-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[Michael Bell]

Josh Guilfoyle schrieb:

> Having fixed that, my second problem still seems to be valid.  In
> certain cases, the server will send an Add command with an incomplete
> item and specify MoreData.  Then, on the next package, it will simply
> send Status and Final, and libsyncml will emit SML_MANAGER_SESSION_END.
>  My client then responds with Alert 222 (NEXT_MESSAGE), and libsyncml
> resumes sending the incomplete item from the original message.
>
> This may be my fault, however, as I think my usage of <Final> is
> incorrect when sending status "213 - Chunk accepted".  See MsgID 5 from
> the client.  Perhaps when I accept an incomplete item I should not mark
> the package as Final?

Perhaps there is a misunderstanding of the word "package". A package is
a logical unit of the SyncML protocol and can consist of several messages.

Example: sync package of the client

The client sync package consists of all sync commands of the client
including all add, replace and delete sub commands. If all these
commands including their data is transmitted then the client MUST send
<final/>. Only after this final the server can send a final (except of
an abort because of an error).

(end of example)

So if their is an active sync package and the last message contains
<MoreData/> then the other peer must answer with status 213 and command
222. A <final/> is only allowed if the other peer aborts the
synchronization because of an error.

So your own diagnose is partly correct. The <final/> is wrong and the
package can only be closed if the remote peer closes its package too.

Best regards

Michael

P.S. please see chapter 2.9 "Multiple Messages in Package" in [1].

>> [1]
>> http://www.openmobilealliance.org/tech/affiliates/syncml/syncml_sync_protocol_v11_20020215.pdf 
--
_______________________________________________________________

Michael Bell                    Humboldt-Universitaet zu Berlin

Tel.: [phone]       ZE Computer- und Medienservice
Fax:  [phone]       Unter den Linden 6
michael.bell@...   D-10099 Berlin
_______________________________________________________________

X.509 CA Certificates / Wurzelzertifikate

http://ra.pki.hu-berlin.de


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel

[jasta]

I fixed the <Final> problem and now my client is communicating
effectively with libsyncml.  Thanks a bunch for your help!

Michael Bell wrote:

> Josh Guilfoyle schrieb:
>> Having fixed that, my second problem still seems to be valid.  In
>> certain cases, the server will send an Add command with an incomplete
>> item and specify MoreData.  Then, on the next package, it will simply
>> send Status and Final, and libsyncml will emit
>> SML_MANAGER_SESSION_END.  My client then responds with Alert 222
>> (NEXT_MESSAGE), and libsyncml resumes sending the incomplete item from
>> the original message.
>>
>> This may be my fault, however, as I think my usage of <Final> is
>> incorrect when sending status "213 - Chunk accepted".  See MsgID 5
>> from the client.  Perhaps when I accept an incomplete item I should
>> not mark the package as Final?
>
> Perhaps there is a misunderstanding of the word "package". A package is
> a logical unit of the SyncML protocol and can consist of several messages.
>
> Example: sync package of the client
>
> The client sync package consists of all sync commands of the client
> including all add, replace and delete sub commands. If all these
> commands including their data is transmitted then the client MUST send
> <final/>. Only after this final the server can send a final (except of
> an abort because of an error).
>
> (end of example)
>
> So if their is an active sync package and the last message contains
> <MoreData/> then the other peer must answer with status 213 and command
> 222. A <final/> is only allowed if the other peer aborts the
> synchronization because of an error.
>
> So your own diagnose is partly correct. The <final/> is wrong and the
> package can only be closed if the remote peer closes its package too.
>
> Best regards
>
> Michael
>
> P.S. please see chapter 2.9 "Multiple Messages in Package" in [1].
>
>>> [1]
>>> http://www.openmobilealliance.org/tech/affiliates/syncml/syncml_sync_protocol_v11_20020215.pdf 
>

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
Opensync-devel mailing list
Opensync-devel@...
https://lists.sourceforge.net/lists/listinfo/opensync-devel