You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Rick Lee-Morlang <ri...@justleapin.com> on 2007/12/19 20:52:48 UTC

repository corruption, many times

Hi all,

(This is a repost, because we just had the same problem, *again*. I've
already tried to post this, but it never made it through the moderation
queue. Sorry if it ends up being duplicated.)

I'm stumped and frustrated.

I'll admit from the outset that we're using Subversion in a non-optimal way.
In part of our repo, we have a group of artists committing binary files,
some of which get rather large. We know this won't scale in the long term,
but there's a difference between SVN being intolerably slow (which it's not)
and corrupting data (which it is).

Our initial setup was on a Fedora 7 machine running svnserve 1.4.3 with an
FSFS backend, integrated with Trac via the standard pre and post-commit hook
scripts. We experienced a few instances of corruption with this setup,
always during large binary commits, and almost always from the same
particular workstation. However in this handful of instances we were always
able to identify the pin the blame on either running out of diskspace or
having mod_svn and svnserve running against the repository at the same time.


To resolve the corruption we restored from a known good backup and then did
a svnadmin dump | svnadmin load for the revs between the known good and the
point of corruption. At that point we committed a change against a
placeholder file so the rev numbers still matched up with Trac, and went on
about our business.

Since then, we've moved SVN to a Centos 5 box running svnserve 1.4.2 where
diskspace is not an issue anymore. All seemed fine for a month and a half,
and then in the last week we've had three corruption instances, two
occurring today. The second happened within 15 minutes of bringing the repo
back up after recovery.

This time I noticed two new details. I feel like I'm grasping at straws,
but:


1) All three of the recent, unexplained corruptions were:
 * large-ish commits mostly consisting of binaries (i.e. at least 10s of MB)

 * that were happening within a very close timespan (likely 5 - 15 seconds)
of another commit (some cases small, one case also large)

This makes me wonder about how svnserve is handling file locking, which led
me to:

2) For some reason we're running svnserve with --threads.



After this I tried to reproduce the problem by creating a new repo and
writing a script that repeated wrote 10MB of random data to a file and
committed it while another script wrote five tiny files of random data and
conmitted those. No corruption.

The only thing I can think of at this point, aside from an outright bug, is
that when the trac pre/post commit scripts call back to check the repository
they're causing corruption -- but afaik they should be read-only ops and
this shouldn't cause any harm.

Any ideas? Should I bump this on to the dev list?

Meanwhile, I'm off to recover the repo. Again.

thanks,
Rick

Re: repository corruption, many times

Posted by Norbert Unterberg <nu...@gmail.com>.
On Dec 19, 2007 10:45 PM, Rick Lee-Morlang
<ri...@justleapin.com> wrote:
>
>
>
> On 12/19/07, Jeremy Whitlock <jc...@gmail.com> wrote:
> > > The only thing I can think of at this point, aside from an outright bug,
[...]

Subversion 1.4.6 has just been released., and the release notes list
this as a server change:
    * fixed: invalid FSFS directory cache can corrupt repository (r27256)

Maybe you give it a try?

Norbert

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: repository corruption, many times

Posted by Rick Lee-Morlang <ri...@justleapin.com>.
On 12/19/07, Jeremy Whitlock <jc...@gmail.com> wrote:
>
> Rick,
>     After looking at your script, can you tell me a few things:
>
> 1) In svn_check_commit, why do you "verify" the current rev and rev -1?


That's leftover from when we were first trying to understand what was
happening. It wasn't clear when corruption was happening, so the script
checked the most recent two revisions on every commit. Now we now it's the
most recent commit that corrupts.

2) When svn_check_commit fails, can you tell me the exit number?


The exit value of *my* script will always be 0 unless it's called with no
arguments.  However, if you're interested in the result of svnadmin dump....
see the next answer

3) Any chance you could get the error coming from Subversion?


[root@thebe subversion_BROKEN_r1297]# svnadmin dump --incremental -r 1297 .
> /dev/null
svnadmin: Decompression of svndiff data failed
[root@thebe subversion_BROKEN_r1297]# echo $?
1

We'll start there.  At first glance, everything appears fine.  I have
> a few ideas about concurrent access to "svnadmin dump" but we'll get
> there eventually.


It would be horribly ironic if the script that's checking for corruption is
the thing that's corrupting the repo.

And, now that you mention it, it *is* the case that this particular kind of
corruption (i.e., inexplicable, thus far) didn't show up until I put that
script in place to try to trap the previous kinds of corruption. I assumed
that svnadmin dump would be a read-only op and would therefore be safe even
with concurrent access.

Rick

Re: repository corruption, many times

Posted by Jeremy Whitlock <jc...@gmail.com>.
Rick,
    After looking at your script, can you tell me a few things:

1) In svn_check_commit, why do you "verify" the current rev and rev -1?
2) When svn_check_commit fails, can you tell me the exit number?
3) Any chance you could get the error coming from Subversion?

We'll start there.  At first glance, everything appears fine.  I have
a few ideas about concurrent access to "svnadmin dump" but we'll get
there eventually.

Take care,

Jeremy

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: repository corruption, many times

Posted by Rick Lee-Morlang <ri...@justleapin.com>.
On 12/19/07, Jeremy Whitlock <jc...@gmail.com> wrote:
>
> > The only thing I can think of at this point, aside from an outright bug,
> is
> > that when the trac pre/post commit scripts call back to check the
> repository
> > they're causing corruption -- but afaik they should be read-only ops and
> > this shouldn't cause any harm.
>
>
> Can you send the pre/post hooks for Trac?


Sure. Will send as another email. They're basically the vanilla scripts that
come with trac with a couple regex bugfixes.

I have Trac+Subversion
> running without any scripts and I'd like to see what you're doing.  As
> for corruption, any error messages or access/error logs would be
> great.


No error logs. I've got a script that checks for corruption on every commit.
The commit that corrupts is always successful, but any subsequent commits to
the same files trigger errors. Access logs are just the SVN logs themselves.
I also have a couple examples of corrupted changeset files in revs. Relevant
bit is:

------------------------------------------------------------------------
r1296 | john | 2007-12-19 11:46:09 -0800 (Wed, 19 Dec 2007) | 1 line

refs#202 updated tracking spreadsheet to reflect latest thumbnail additions
------------------------------------------------------------------------
r1297 | joelp | 2007-12-19 11:46:15 -0800 (Wed, 19 Dec 2007) | 1 line

ART: Props: accessories: textures: globes, easels, umbrella; refs#195
------------------------------------------------------------------------

1297 is the one that ended up being corrupted today. Note the proximity of
the timestamps. 1296 was a 17K changeset, 1297 was a 25MB changeset.

  Are you using only svnserve or are you still doing Apache and
> svnserve?


Only using svnserve at this point.

Rick

Re: repository corruption, many times

Posted by Jeremy Whitlock <jc...@gmail.com>.
> The only thing I can think of at this point, aside from an outright bug, is
> that when the trac pre/post commit scripts call back to check the repository
> they're causing corruption -- but afaik they should be read-only ops and
> this shouldn't cause any harm.


Can you send the pre/post hooks for Trac?  I have Trac+Subversion
running without any scripts and I'd like to see what you're doing.  As
for corruption, any error messages or access/error logs would be
great.  Are you using only svnserve or are you still doing Apache and
svnserve?

Take care,

Jeremy

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org