Corrupt svndiff data committed to repository

[Chris Pearson-3]

Hi,

We've recently discovered a problem with subversion.  When a particular
file we have is committed to a subversion repository, sometimes after
committing the repository is left in a corrupt state.  It's almost as
if there's a magic sequence of bytes that svnkit struggles with and
sends corrupt svndiff data to the subversion server.  There is a
corresponding bug in subversion such that it allows svnkit to
successfully commit a corrupt revision, but we haven't tried raising it
with them yet.  Following the commit, svnadmin verify reports
"svnadmin: Decompression of svndiff data failed".

We know:
 * it only affects svnkit clients (IDEA and SmartSVN both cause it)
 * it doesn't seem to affect the command line subversion c clients
 * it only affects commits involving a particular file (mail me off list
   if you'd like a copy, it's about 34MB)
 * a file of dummy data and the same size doesn't cause the issue
 * the subversion repository is left with intact windows, but the zlib
   compressed data in one of those windows is not of a valid format
 * it only seems to affect svn over http/https

We can reproduce this, but I'm unsure where to take it from here.  Can
anyone give some pointers on how we should debug further?  If you like
we can corrupt a repository if someone wants to see for themselves?

Thanks,
Chris

---------------------------------------------------------------------
To unsubscribe, e-mail: svnkit-users-unsubscribe@...
For additional commands, e-mail: svnkit-users-help@...

[Alexander Kitaev-3]

Hello Chris,

 > We can reproduce this, but I'm unsure where to take it from here.  Can
 > anyone give some pointers on how we should debug further?  If you like
 > we can corrupt a repository if someone wants to see for themselves?
It will help if you'll provide "magic" file to let us reproduce the
problem as well. If you could put it somewhere, please send me a link to
the file to Alexander.Kitaev@...

Thanks!

Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!

Chris Pearson wrote:

> Hi,
>
> We've recently discovered a problem with subversion.  When a particular
> file we have is committed to a subversion repository, sometimes after
> committing the repository is left in a corrupt state.  It's almost as
> if there's a magic sequence of bytes that svnkit struggles with and
> sends corrupt svndiff data to the subversion server.  There is a
> corresponding bug in subversion such that it allows svnkit to
> successfully commit a corrupt revision, but we haven't tried raising it
> with them yet.  Following the commit, svnadmin verify reports
> "svnadmin: Decompression of svndiff data failed".
>
> We know:
>  * it only affects svnkit clients (IDEA and SmartSVN both cause it)
>  * it doesn't seem to affect the command line subversion c clients
>  * it only affects commits involving a particular file (mail me off list
>    if you'd like a copy, it's about 34MB)
>  * a file of dummy data and the same size doesn't cause the issue
>  * the subversion repository is left with intact windows, but the zlib
>    compressed data in one of those windows is not of a valid format
>  * it only seems to affect svn over http/https
>
> We can reproduce this, but I'm unsure where to take it from here.  Can
> anyone give some pointers on how we should debug further?  If you like
> we can corrupt a repository if someone wants to see for themselves?
>
> Thanks,
> Chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: svnkit-users-unsubscribe@...
> For additional commands, e-mail: svnkit-users-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: svnkit-users-unsubscribe@...
For additional commands, e-mail: svnkit-users-help@...

[Alexander Kitaev-3]

Hello Chris,

I've downloaded the file for which corruption problem occurs and will
try to reproduce the problem with my test repository.

However, there is something that makes me think that the problem is not
in delta generation algorithm:

 >  * it only seems to affect svn over http/https
The same code is used to generate delta for all protocols. So, if
corruption error occurs only for http, then the reason of the problem is
not in delta code.

 >  * a file of dummy data and the same size doesn't cause the issue
There is a difference still - delta for a dummy data (e.g. all zeroes)
will be different from the one for the real file - it will be much smaller.

Are you sure that repository indeed get corrupted and it is not a
checkout/update problem? It could happen that commit succeeds and it is
update or checkout that then fails on that repository and particular
working copy.

If it is the case, could you please try enabling spooling for all HTTP
operations:

DAVRepositoryFactory.setup(
    new DefaultHTTPConnectionFactory(null, true, null));

What version of SVNKit do you use?

Alexander Kitaev,
TMate Software,
http://svnkit.com/ - Java [Sub]Versioning Library!

Chris Pearson wrote:

> Hi,
>
> We've recently discovered a problem with subversion.  When a particular
> file we have is committed to a subversion repository, sometimes after
> committing the repository is left in a corrupt state.  It's almost as
> if there's a magic sequence of bytes that svnkit struggles with and
> sends corrupt svndiff data to the subversion server.  There is a
> corresponding bug in subversion such that it allows svnkit to
> successfully commit a corrupt revision, but we haven't tried raising it
> with them yet.  Following the commit, svnadmin verify reports
> "svnadmin: Decompression of svndiff data failed".
>
> We know:
>  * it only affects svnkit clients (IDEA and SmartSVN both cause it)
>  * it doesn't seem to affect the command line subversion c clients
>  * it only affects commits involving a particular file (mail me off list
>    if you'd like a copy, it's about 34MB)
>  * a file of dummy data and the same size doesn't cause the issue
>  * the subversion repository is left with intact windows, but the zlib
>    compressed data in one of those windows is not of a valid format
>  * it only seems to affect svn over http/https
>
> We can reproduce this, but I'm unsure where to take it from here.  Can
> anyone give some pointers on how we should debug further?  If you like
> we can corrupt a repository if someone wants to see for themselves?
>
> Thanks,
> Chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: svnkit-users-unsubscribe@...
> For additional commands, e-mail: svnkit-users-help@...
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: svnkit-users-unsubscribe@...
For additional commands, e-mail: svnkit-users-help@...

[Chris Pearson-3]

Hi there,

On Tue, Jul 29, 2008 at 03:21:48PM +0200, Alexander Kitaev wrote:
> >  * it only seems to affect svn over http/https
> The same code is used to generate delta for all protocols. So, if corruption error occurs only for http, then the reason of the problem is not in delta code.

OK.  The end result is that the repository has one or more corrupt delta
windows.  I am unsure whether svnkit is sending a corrupt window.  This
could be almost entirely a subversion bug, but somehow triggered by
something svnkit does differently from their c client.

> >  * a file of dummy data and the same size doesn't cause the issue
> There is a difference still - delta for a dummy data (e.g. all zeroes) will be different from the one for the real file - it will be much smaller.
>
> Are you sure that repository indeed get corrupted and it is not a checkout/update problem? It could happen that commit succeeds and it is update or checkout
> that then fails on that repository and particular working copy.

Absolutely positive.  svnadmin verify fails after it has been corrupted,
and on further investigation into the repository's db/revs file, the
latest one contains invalid zlib data in one of the delta windows.
If you do try a clean checkout, it reports a failure in the REPORT call
to the client, and they can't check out.

Cheers,
Chris

---------------------------------------------------------------------
To unsubscribe, e-mail: svnkit-users-unsubscribe@...
For additional commands, e-mail: svnkit-users-help@...