You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by "Andersen, Krista" <Kr...@itg.com> on 2010/01/13 00:28:15 UTC
sync bug -> corrupted proxy repo
Twice I have seen one of my proxy repositories become corrupted due to an apparent bug in the svnsync sync process. Has anyone else seen this type of behavior from Subversion?
I am able to move the corrupted proxy-repo and recreate it again without error - but I am a bit concerned about the stability of Subversion since this is the second time in two months that I have had to fix this issue.
Here is a comparison the output of the svn log -v for the offending revisions (324,325) on both the corrupted and non-corrupted proxy repo. ***Notice a different author, earlier time for later rev, different action, different path, and different rev copied from***
[svnadmin@SUBSERVER:/home/Svn/repos/OPT]> svn log -r 323:325 -v http://subserver/OPT/OPT.corrupted
------------------------------------------------------------------------
r323 | optauto | 2010-01-06 06:22:07 +0200 (Wed, 06 Jan 2010) | 1 line
Changed paths:
A /tags/ITR_5.4.0.122/OptBackEnd (from /trunk/OptBackEnd:322)
Iteration 5.4.0.122 label.
------------------------------------------------------------------------
r324 | svnservice | 2010-01-06 05:24:01 +0200 (Wed, 06 Jan 2010) | 1 line
Changed paths:
R /tags/ITR_5.4.0.122/OptBackEnd (from /trunk/OptBackEnd:322)
Iteration 5.4.0.122 label.
------------------------------------------------------------------------
r325 | svnservice | 2010-01-06 05:24:08 +0200 (Wed, 06 Jan 2010) | 1 line
Changed paths:
A /tags/ITR_5.4.0.122/OptFrontEnd (from /trunk/OptFrontEnd:323)
Iteration 5.4.0.122 label.
------------------------------------------------------------------------
[svnadmin@SUBSERVER:/home/Svn/repos/OPT]> svn log -r 323:325 -v http://subserver/OPT/OPT
------------------------------------------------------------------------
r323 | optauto | 2010-01-06 06:22:07 +0200 (Wed, 06 Jan 2010) | 1 line
Changed paths:
A /tags/ITR_5.4.0.122/OptBackEnd (from /trunk/OptBackEnd:322)
Iteration 5.4.0.122 label.
------------------------------------------------------------------------
r324 | optauto | 2010-01-06 06:22:08 +0200 (Wed, 06 Jan 2010) | 1 line
Changed paths:
A /tags/ITR_5.4.0.122/OptFrontEnd (from /trunk/OptFrontEnd:323)
Iteration 5.4.0.122 label.
------------------------------------------------------------------------
r325 | optauto | 2010-01-06 06:22:09 +0200 (Wed, 06 Jan 2010) | 1 line
Changed paths:
A /tags/ITR_5.4.0.122/OPT (from /trunk/OPT:324)
Iteration 5.4.0.122 label.
------------------------------------------------------------------------
Also, here is a comparison the output of the svn log -v for more offending revisions (49,50) on both the corrupted and non-corrupted proxy repo. ***Notice again, the different authors, earlier time for later rev, different actions, different path, and different rev copied from***
[svnadmin@SUBSERVER:/home/Svn/repos/OPT]> svn log -r 48:50 -v http://subserver/OPT/OPT.corrupted
------------------------------------------------------------------------
r48 | optauto | 2009-11-09 19:19:19 +0200 (Mon, 09 Nov 2009) | 1 line
Changed paths:
A /tags/ITR_5.4.0.118/OptBackEnd (from /trunk/OptBackEnd:47)
Iteration 5.4.0.118 label.
------------------------------------------------------------------------
R49 | svnservice | 2009-11-09 18:19:03 +0200 (Mon, 09 Nov 2009) | 1 line
Changed paths:
R /tags/ITR_5.4.0.118/OptBackEnd (from /trunk/OptBackEnd:47)
Iteration 5.4.0.118 label.
------------------------------------------------------------------------
R50 | svnservice | 2009-11-09 18:19:12 +0200 (Mon, 09 Nov 2009) | 1 line
Changed paths:
A /tags/ITR_5.4.0.118/OptFrontEnd (from /trunk/OptFrontEnd:48)
Iteration 5.4.0.118 label.
------------------------------------------------------------------------
[svnadmin@SUBSERVER:/home/Svn/repos/OPT]> svn log -r 48:50 -v http://subserver/OPT/OPT
------------------------------------------------------------------------
r48 | optauto | 2009-11-09 19:19:19 +0200 (Mon, 09 Nov 2009) | 1 line
Changed paths:
A /tags/ITR_5.4.0.118/OptBackEnd (from /trunk/OptBackEnd:47)
Iteration 5.4.0.118 label.
------------------------------------------------------------------------
r49 | optauto | 2009-11-09 19:19:20 +0200 (Mon, 09 Nov 2009) | 1 line
Changed paths:
A /tags/ITR_5.4.0.118/OptFrontEnd (from /trunk/OptFrontEnd:48)
Iteration 5.4.0.118 label.
------------------------------------------------------------------------
r50 | optauto | 2009-11-09 19:19:21 +0200 (Mon, 09 Nov 2009) | 1 line
Changed paths:
A /tags/ITR_5.4.0.118/OPT (from /trunk/OPT:49)
Iteration 5.4.0.118 label.
------------------------------------------------------------------------
My master repo is on sparc-solaris2.10, apache2.2.10, Subversion 1.6.3 and is physically located in NY. I use standard post-commit hook script to launch the svnsync sync command after each commit. My proxy repo is x86-solaris2.10 and is physically located in Tel Aviv.
The user shown in the corrupted revs is our admin account. The other actions just before and just after the corrupted revs show a generic user for our auto-build tool(ANT).
Has anyone else seen corrupted syncs before? Is there a patch or something to prevent this in the future?
Thank you,
Krista
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
This message is for the named person's use only. This communication is for
informational purposes only and has been obtained from sources believed to
be reliable, but it is not necessarily complete and its accuracy cannot be
guaranteed. It is not intended as an offer or solicitation for the purchase
or sale of any financial instrument or as an official confirmation of any
transaction. Moreover, this material should not be construed to contain any
recommendation regarding, or opinion concerning, any security. It may
contain confidential, proprietary or legally privileged information. No
confidentiality or privilege is waived or lost by any mistransmission. If
you receive this message in error, please immediately delete it and all
copies of it from your system, destroy any hard copies of it and notify the
sender. You must not, directly or indirectly, use, disclose, distribute,
print, or copy any part of this message if you are not the intended
recipient. Any views expressed in this message are those of the individual
sender, except where the message states otherwise and the sender is
authorized to state them to be the views of any such entity.
Securities products and services provided to Canadian investors are offered
by ITG Canada Corp. (member CIPF and IIROC - Investment Industry Regulatory
Organization of Canada), an affiliate of Investment
Technology Group, Inc.
ITG Inc. and/or its affiliates reserves the right to monitor and archive
all electronic communications through its network.
ITG Inc. Member FINRA, SIPC
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
Re: sync bug -> corrupted proxy repo
Posted by Johan Corveleyn <jc...@gmail.com>.
On Fri, Jan 15, 2010 at 12:08 PM, Jon Foster <Jo...@cabot.co.uk> wrote:
> Hi,
>
> Ryan Schmidt wrote:
>> But Subversion blocks the commit until the post-commit is done.
>
> That particular SVN client will be blocked. But if you have
> two users committing at the same time, or if a user runs "svn"
> twice in parallel, then the post-commit hook will be run in
> parallel.
>
> Here's how I tested this. I created a new repository with
> a post-commit hook that takes 30 seconds to run. I then
> checked that it works, and that a normal commit took 30
> seconds. I then did two commits in parallel, and that took
> 30 seconds. This shows that the post-commit hook is
> running in parallel - if it had been run in series, then
> it would have taken 60 seconds for 2 commits. (I also
> checked the output of "ps" and observed the two
> "post-commit" processes running).
Also, I'm pretty sure that, while the post-commit hook is running for
a particular commit, the commit itself is already visible to other
users. So, as you would expect from the name "*post*-commit hook", the
commit itself is already finalized before the post-commit hook starts
running. Otherwise, people wouldn't be able to do things like
automatically updating a working copy on the server, from within their
post-commit hook. The only thing that has to wait on the post-commit
hook is that particular svn client that's running the commit (as Jon
pointed out).
Regards,
Johan
RE: sync bug -> corrupted proxy repo
Posted by Jon Foster <Jo...@cabot.co.uk>.
Hi,
Ryan Schmidt wrote:
> But Subversion blocks the commit until the post-commit is done.
That particular SVN client will be blocked. But if you have
two users committing at the same time, or if a user runs "svn"
twice in parallel, then the post-commit hook will be run in
parallel.
Here's how I tested this. I created a new repository with
a post-commit hook that takes 30 seconds to run. I then
checked that it works, and that a normal commit took 30
seconds. I then did two commits in parallel, and that took
30 seconds. This shows that the post-commit hook is
running in parallel - if it had been run in series, then
it would have taken 60 seconds for 2 commits. (I also
checked the output of "ps" and observed the two
"post-commit" processes running).
~$ mkdir svnscratch
~$ cd svnscratch/
~/svnscratch$ svn --version | head -n1
svn, version 1.6.8 (dev build)
~/svnscratch$ svnadmin create repo
~/svnscratch$ cat >repo/hooks/post-commit
#! /bin/bash
sleep 30
~/svnscratch$ chmod a+x repo/hooks/post-commit
~/svnscratch$ time repo/hooks/post-commit
real 0m30.004s
user 0m0.000s
sys 0m0.008s
~/svnscratch$ time svn mkdir -m "Test" file://`pwd`/repo/trunk
Committed revision 1.
real 0m30.030s
user 0m0.008s
sys 0m0.008s
~/svnscratch$ time ( svn mkdir -m "Test" file://`pwd`/repo/branches &
svn mkdir -m "Test" file://`pwd`/repo/tags )
Committed revision 2.
Committed revision 3.
real 0m30.069s
user 0m0.004s
sys 0m0.020s
~/svnscratch$
Kind regards,
Jon
**********************************************************************
This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Cabot Communications Ltd.
If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone.
Cabot Communications Limited
Verona House, Filwood Road, Bristol BS16 3RY, UK
+44 (0) 1179584232
Co. Registered in England number 02817269
Please contact the sender if you believe you have received this email in error.
**********************************************************************
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Re: sync bug -> corrupted proxy repo
Posted by Ryan Schmidt <su...@ryandesign.com>.
On Jan 14, 2010, at 09:06, Johan Corveleyn wrote:
>> Is it not the case that a svn commit cannot start before the post-commit hooks has finished? I am asking becuase I will be implementing a DR system using svnsync, but I am not planning to let the post-commit finish before svnsync has finished (I don't care if it takes a bit longer, I can cope with that and with my users).
>
> No, that's definitely not the case. Both pre-commit and post-commit
> hooks can run simultaneously for multiple commits in parallel.
> Otherwise, a pre/post-commit hook could end up being a big bottleneck.
>
> As a practical example: if your pre-commit hook does validation of
> properties (e.g. making sure svn:eol-style is set correctly on the
> right types of files), then for a commit with a lot of files it may be
> running for multiple seconds (even minutes). In the meantime, other
> commits can be made without problems, without being bothered by that
> one commit which takes a long time to pass through pre-commit hook. I
> saw this in action myself with such a pre-commit hook, after I added
> some debug logging at the start and the end of the hook.
pre-commit hook, I believe you're right. But Subversion blocks the commit until the post-commit is done. Only one post-commit hook will run at a time. Unless you tell it to allow simultaneous runs, by redirecting the hook's stdout and stderr someplace.
Re: sync bug -> corrupted proxy repo
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Jan 13, 2010 at 2:27 PM, Giulio Troccoli
<Gi...@uk.linedata.com> wrote:
>> From: Jon Foster [mailto:Jon.Foster@cabot.co.uk]
>> Sent: 13 January 2010 13:13
>> To: Andersen, Krista; users@subversion.apache.org
>> Cc: ssi-svn_admin
>> Subject: RE: sync bug -> corrupted proxy repo
>>
>> Hi,
>>
>> Andersen, Krista [mailto:Krista.Andersen@itg.com] wrote:
>> > Twice I have seen one of my proxy repositories become
>> corrupted due to
>> > an apparent bug in the svnsync sync process. Has anyone else seen
>> > this type of behavior from Subversion?
>>
>> This is probably caused by issue 3546 [1][2]. This is a race
>> condition - if you have several sync processes running at the
>> same time then the mirror can get corrupted. You had three
>> commits which were 1 second apart, so your hook script
>> started 3 copies of svnsync within 2 seconds. I think this
>> is the first practical report of this bug; previous
>> discussion was theoretical.
>>
>> > Here is a comparison the output of the svn log -v for the offending
>> > revisions (324,325) on both the corrupted and non-corrupted proxy
>> > repo.
>>
>> It looks like rev 323 got mirrored twice (as mirror revs 323
>> and 324), then rev 324 was mirrored (as mirror rev 325).
>>
>> > I am a bit concerned about the stability of Subversion
>> since this is
>> > the second time in two months that I have had to fix this issue.
>> > Is there a patch or something to prevent this in the future?
>>
>> Suggested workaround: Change your hook scripts to use the
>> lockf or lockfile tools[3] to ensure that only one instance
>> of svnsync runs at once.
>
> Is it not the case that a svn commit cannot start before the post-commit hooks has finished? I am asking becuase I will be implementing a DR system using svnsync, but I am not planning to let the post-commit finish before svnsync has finished (I don't care if it takes a bit longer, I can cope with that and with my users).
No, that's definitely not the case. Both pre-commit and post-commit
hooks can run simultaneously for multiple commits in parallel.
Otherwise, a pre/post-commit hook could end up being a big bottleneck.
As a practical example: if your pre-commit hook does validation of
properties (e.g. making sure svn:eol-style is set correctly on the
right types of files), then for a commit with a lot of files it may be
running for multiple seconds (even minutes). In the meantime, other
commits can be made without problems, without being bothered by that
one commit which takes a long time to pass through pre-commit hook. I
saw this in action myself with such a pre-commit hook, after I added
some debug logging at the start and the end of the hook.
Regards,
Johan
RE: sync bug -> corrupted proxy repo
Posted by Giulio Troccoli <Gi...@uk.linedata.com>.
>
Linedata Services (UK) Ltd
Registered Office: Bishopsgate Court, 4-12 Norton Folgate, London, E1 6DB
Registered in England and Wales No 3027851 VAT Reg No 778499447
-----Original Message-----
> From: Jon Foster [mailto:Jon.Foster@cabot.co.uk]
> Sent: 13 January 2010 13:13
> To: Andersen, Krista; users@subversion.apache.org
> Cc: ssi-svn_admin
> Subject: RE: sync bug -> corrupted proxy repo
>
> Hi,
>
> Andersen, Krista [mailto:Krista.Andersen@itg.com] wrote:
> > Twice I have seen one of my proxy repositories become
> corrupted due to
> > an apparent bug in the svnsync sync process. Has anyone else seen
> > this type of behavior from Subversion?
>
> This is probably caused by issue 3546 [1][2]. This is a race
> condition - if you have several sync processes running at the
> same time then the mirror can get corrupted. You had three
> commits which were 1 second apart, so your hook script
> started 3 copies of svnsync within 2 seconds. I think this
> is the first practical report of this bug; previous
> discussion was theoretical.
>
> > Here is a comparison the output of the svn log -v for the offending
> > revisions (324,325) on both the corrupted and non-corrupted proxy
> > repo.
>
> It looks like rev 323 got mirrored twice (as mirror revs 323
> and 324), then rev 324 was mirrored (as mirror rev 325).
>
> > I am a bit concerned about the stability of Subversion
> since this is
> > the second time in two months that I have had to fix this issue.
> > Is there a patch or something to prevent this in the future?
>
> Suggested workaround: Change your hook scripts to use the
> lockf or lockfile tools[3] to ensure that only one instance
> of svnsync runs at once.
Is it not the case that a svn commit cannot start before the post-commit hooks has finished? I am asking becuase I will be implementing a DR system using svnsync, but I am not planning to let the post-commit finish before svnsync has finished (I don't care if it takes a bit longer, I can cope with that and with my users).
Giulio
RE: sync bug -> corrupted proxy repo
Posted by Jon Foster <Jo...@cabot.co.uk>.
Hi,
> I am assuming that if the commits must start at least one
> second apart - then the sync from the post-commit hook
> would not be able to reach a race condition.
> Is this a reasonable assumption?
No, the bug is worse than that. Suppose there are 3 commits:
- At time 12:00:00, a commit starts sync process #1. The sync
takes 6 seconds.
- At time 12:00:02, a commit starts sync process #2. This blocks
due to sync process #1's lock.
- At time 12:00:04, a commit starts sync process #3. This blocks
due to sync process #1's lock.
- At time 12:00:06, sync process #1 finishes. Sync processes #2 and
#3 both try to take the lock; due to the bug they may _both_
succeed in taking the lock. Chaos ensues.
I suggest you use the "flock(1)" tool. [1]. This is installed as
a standard part of Debian (it's in the util-linux package).
Something like this, in your post-commit hook:
--- cut here - start ---
#!/bin/sh
/usr/bin/flock --wait 1200 \
-x /var/lock/svn_sync_lock \
/usr/local/bin/svnsync sync --non-interactive \
http://mirrorserver.example.com/svn &
--- cut here - end ---
You will need to make the /var/lock/svn_sync_lock file and ensure
it's writable by the user your post-commit hook is running as.
"flock" is a mature, tested piece of code to handle locking.
It will ensure that only one copy of svnsync is running at a
time. That way, the race condition in svnsync is avoided.
Kind regards,
Jon
[1] http://www.google.co.uk/search?q=man+flock%281%29
-----Original Message-----
From: Andersen, Krista [mailto:Krista.Andersen@itg.com]
Sent: 15 January 2010 22:29
To: Jon Foster; users@subversion.apache.org
Cc: ssi-svn_admin
Subject: RE: sync bug -> corrupted proxy repo
Thank you Jon, for your explanation and workaround.
Are there any "best practices" that we can advise our dev groups to
follow to avoid this problem?
Otherwise, your suggestions seem to indicate I would have to run the
sync on a cronjob and not with the hook script. That is something we
would like to avoid. So I have added a start time comparison and sleep
in a start-commit hook instead. Do you see any reason why this would
cause other problems?
I am assuming that if the commits must start at least one second apart -
then the sync from the post-commit hook would not be able to reach a
race condition. Is this a reasonable assumption?
#!/usr/bin/sh
# START-COMMIT HOOK
# kanderse Jan 13, 2010
# The start-commit hook is invoked before a Subversion txn is created
# in the process of doing a commit.
# This script checks the start time and compares with the start time
# of the previous commit. It will cause a commit to wait one second if
# the last commit was started less than one second earlier.
# The purpose of this wait is to prevent known issue 3546 [1][2].
# a race condition involving multiple sync processes running at
# the same time that result in a corrupted proxy.
REPOS="$1"
USER="$2"
DATE1=`cat /$REPOS/hooks/start-time.txt` # previous start time
DATE2=`/usr/local/bin/date +%s` # record current start time
echo $DATE2 > /$REPOS/hooks/start-time.txt
# echo $DATE2 $DATE1 `expr $DATE2 - $DATE1`
if [ `expr $DATE2 - $DATE1` -lt 1 ]
then sleep 1 # to prevent sync race that results in sync duplication and
corrupted proxy
fi
# All checks passed, so allow the commit.
exit 0
Krista Andersen
Global Development Infrastructure
Investment Technology Group, Inc.
400 Corporate Pointe, 8th Floor
Culver City, CA 90230
Direct: 213.270.7570
-----Original Message-----
From: Jon Foster [mailto:Jon.Foster@cabot.co.uk]
Sent: Wednesday, January 13, 2010 5:13 AM
To: Andersen, Krista; users@subversion.apache.org
Cc: ssi-svn_admin
Subject: RE: sync bug -> corrupted proxy repo
Hi,
Andersen, Krista [mailto:Krista.Andersen@itg.com] wrote:
> Twice I have seen one of my proxy repositories become corrupted due
> to an apparent bug in the svnsync sync process. Has anyone else
> seen this type of behavior from Subversion?
This is probably caused by issue 3546 [1][2]. This is a race
condition - if you have several sync processes running at the same
time then the mirror can get corrupted. You had three commits which
were 1 second apart, so your hook script started 3 copies of svnsync
within 2 seconds. I think this is the first practical report of this
bug; previous discussion was theoretical.
> Here is a comparison the output of the svn log -v for the offending
> revisions (324,325) on both the corrupted and non-corrupted proxy
> repo.
It looks like rev 323 got mirrored twice (as mirror revs 323 and 324),
then rev 324 was mirrored (as mirror rev 325).
> I am a bit concerned about the stability of Subversion since this
> is the second time in two months that I have had to fix this issue.
> Is there a patch or something to prevent this in the future?
Suggested workaround: Change your hook scripts to use the lockf or
lockfile tools[3] to ensure that only one instance of svnsync runs
at once.
Kind regards,
Jon
[1]
http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2
0091127115356.GC9302@jack.stsp.name%3E
[2] http://subversion.tigris.org/issues/show_bug.cgi?id=3546
[3]
http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2
0091127132659.GE9302@jack.stsp.name%3E
**********************************************************************
This email and its attachments may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views
or opinions expressed are solely those of the author and do not
necessarily represent those of Cabot Communications Ltd.
If you are not the intended recipient of this email and its attachments,
you must take no action based upon them, nor must you copy or show them
to anyone.
Cabot Communications Limited
Verona House, Filwood Road, Bristol BS16 3RY, UK
+44 (0) 1179584232
Co. Registered in England number 02817269
Please contact the sender if you believe you have received this email in
error.
**********************************************************************
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-+-
This message is for the named person's use only. This communication is
for
informational purposes only and has been obtained from sources believed
to
be reliable, but it is not necessarily complete and its accuracy cannot
be
guaranteed. It is not intended as an offer or solicitation for the
purchase
or sale of any financial instrument or as an official confirmation of
any
transaction. Moreover, this material should not be construed to contain
any
recommendation regarding, or opinion concerning, any security. It may
contain confidential, proprietary or legally privileged information. No
confidentiality or privilege is waived or lost by any mistransmission.
If
you receive this message in error, please immediately delete it and all
copies of it from your system, destroy any hard copies of it and notify
the
sender. You must not, directly or indirectly, use, disclose, distribute,
print, or copy any part of this message if you are not the intended
recipient. Any views expressed in this message are those of the
individual
sender, except where the message states otherwise and the sender is
authorized to state them to be the views of any such entity.
Securities products and services provided to Canadian investors are
offered
by ITG Canada Corp. (member CIPF and IIROC - Investment Industry
Regulatory
Organization of Canada), an affiliate of Investment
Technology Group, Inc.
ITG Inc. and/or its affiliates reserves the right to monitor and archive
all electronic communications through its network.
ITG Inc. Member FINRA, SIPC
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
-+-
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
**********************************************************************
This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Cabot Communications Ltd.
If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone.
Cabot Communications Limited
Verona House, Filwood Road, Bristol BS16 3RY, UK
+44 (0) 1179584232
Co. Registered in England number 02817269
Please contact the sender if you believe you have received this email in error.
**********************************************************************
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
RE: sync bug -> corrupted proxy repo
Posted by "Andersen, Krista" <Kr...@itg.com>.
Thank you Jon, for your explanation and workaround.
Are there any "best practices" that we can advise our dev groups to follow to avoid this problem?
Otherwise, your suggestions seem to indicate I would have to run the sync on a cronjob and not with the hook script. That is something we would like to avoid. So I have added a start time comparison and sleep in a start-commit hook instead. Do you see any reason why this would cause other problems?
I am assuming that if the commits must start at least one second apart - then the sync from the post-commit hook would not be able to reach a race condition. Is this a reasonable assumption?
#!/usr/bin/sh
# START-COMMIT HOOK
# kanderse Jan 13, 2010
# The start-commit hook is invoked before a Subversion txn is created
# in the process of doing a commit.
# This script checks the start time and compares with the start time
# of the previous commit. It will cause a commit to wait one second if
# the last commit was started less than one second earlier.
# The purpose of this wait is to prevent known issue 3546 [1][2].
# a race condition involving multiple sync processes running at
# the same time that result in a corrupted proxy.
REPOS="$1"
USER="$2"
DATE1=`cat /$REPOS/hooks/start-time.txt` # previous start time
DATE2=`/usr/local/bin/date +%s` # record current start time
echo $DATE2 > /$REPOS/hooks/start-time.txt
# echo $DATE2 $DATE1 `expr $DATE2 - $DATE1`
if [ `expr $DATE2 - $DATE1` -lt 1 ]
then sleep 1 # to prevent sync race that results in sync duplication and corrupted proxy
fi
# All checks passed, so allow the commit.
exit 0
Krista Andersen
Global Development Infrastructure
Investment Technology Group, Inc.
400 Corporate Pointe, 8th Floor
Culver City, CA 90230
Direct: 213.270.7570
-----Original Message-----
From: Jon Foster [mailto:Jon.Foster@cabot.co.uk]
Sent: Wednesday, January 13, 2010 5:13 AM
To: Andersen, Krista; users@subversion.apache.org
Cc: ssi-svn_admin
Subject: RE: sync bug -> corrupted proxy repo
Hi,
Andersen, Krista [mailto:Krista.Andersen@itg.com] wrote:
> Twice I have seen one of my proxy repositories become corrupted due
> to an apparent bug in the svnsync sync process. Has anyone else
> seen this type of behavior from Subversion?
This is probably caused by issue 3546 [1][2]. This is a race
condition - if you have several sync processes running at the same
time then the mirror can get corrupted. You had three commits which
were 1 second apart, so your hook script started 3 copies of svnsync
within 2 seconds. I think this is the first practical report of this
bug; previous discussion was theoretical.
> Here is a comparison the output of the svn log -v for the offending
> revisions (324,325) on both the corrupted and non-corrupted proxy
> repo.
It looks like rev 323 got mirrored twice (as mirror revs 323 and 324),
then rev 324 was mirrored (as mirror rev 325).
> I am a bit concerned about the stability of Subversion since this
> is the second time in two months that I have had to fix this issue.
> Is there a patch or something to prevent this in the future?
Suggested workaround: Change your hook scripts to use the lockf or
lockfile tools[3] to ensure that only one instance of svnsync runs
at once.
Kind regards,
Jon
[1]
http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2
0091127115356.GC9302@jack.stsp.name%3E
[2] http://subversion.tigris.org/issues/show_bug.cgi?id=3546
[3]
http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2
0091127132659.GE9302@jack.stsp.name%3E
**********************************************************************
This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Cabot Communications Ltd.
If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone.
Cabot Communications Limited
Verona House, Filwood Road, Bristol BS16 3RY, UK
+44 (0) 1179584232
Co. Registered in England number 02817269
Please contact the sender if you believe you have received this email in error.
**********************************************************************
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
This message is for the named person's use only. This communication is for
informational purposes only and has been obtained from sources believed to
be reliable, but it is not necessarily complete and its accuracy cannot be
guaranteed. It is not intended as an offer or solicitation for the purchase
or sale of any financial instrument or as an official confirmation of any
transaction. Moreover, this material should not be construed to contain any
recommendation regarding, or opinion concerning, any security. It may
contain confidential, proprietary or legally privileged information. No
confidentiality or privilege is waived or lost by any mistransmission. If
you receive this message in error, please immediately delete it and all
copies of it from your system, destroy any hard copies of it and notify the
sender. You must not, directly or indirectly, use, disclose, distribute,
print, or copy any part of this message if you are not the intended
recipient. Any views expressed in this message are those of the individual
sender, except where the message states otherwise and the sender is
authorized to state them to be the views of any such entity.
Securities products and services provided to Canadian investors are offered
by ITG Canada Corp. (member CIPF and IIROC - Investment Industry Regulatory
Organization of Canada), an affiliate of Investment
Technology Group, Inc.
ITG Inc. and/or its affiliates reserves the right to monitor and archive
all electronic communications through its network.
ITG Inc. Member FINRA, SIPC
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
RE: sync bug -> corrupted proxy repo
Posted by Jon Foster <Jo...@cabot.co.uk>.
Hi,
Andersen, Krista [mailto:Krista.Andersen@itg.com] wrote:
> Twice I have seen one of my proxy repositories become corrupted due
> to an apparent bug in the svnsync sync process. Has anyone else
> seen this type of behavior from Subversion?
This is probably caused by issue 3546 [1][2]. This is a race
condition - if you have several sync processes running at the same
time then the mirror can get corrupted. You had three commits which
were 1 second apart, so your hook script started 3 copies of svnsync
within 2 seconds. I think this is the first practical report of this
bug; previous discussion was theoretical.
> Here is a comparison the output of the svn log -v for the offending
> revisions (324,325) on both the corrupted and non-corrupted proxy
> repo.
It looks like rev 323 got mirrored twice (as mirror revs 323 and 324),
then rev 324 was mirrored (as mirror rev 325).
> I am a bit concerned about the stability of Subversion since this
> is the second time in two months that I have had to fix this issue.
> Is there a patch or something to prevent this in the future?
Suggested workaround: Change your hook scripts to use the lockf or
lockfile tools[3] to ensure that only one instance of svnsync runs
at once.
Kind regards,
Jon
[1]
http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2
0091127115356.GC9302@jack.stsp.name%3E
[2] http://subversion.tigris.org/issues/show_bug.cgi?id=3546
[3]
http://mail-archives.apache.org/mod_mbox/subversion-dev/200911.mbox/%3C2
0091127132659.GE9302@jack.stsp.name%3E
**********************************************************************
This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Cabot Communications Ltd.
If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone.
Cabot Communications Limited
Verona House, Filwood Road, Bristol BS16 3RY, UK
+44 (0) 1179584232
Co. Registered in England number 02817269
Please contact the sender if you believe you have received this email in error.
**********************************************************************
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________