You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Tony Butt <tj...@cea.com.au> on 2012/08/21 06:54:59 UTC
svnsync fails on a 'pull' operation
We synchronise our production repository to a backup server, using a
push model from post-commit and post-revprop hooks, and also a pull
model (from the backup server) to a different synchronised copy as an
hourly cron job.
We also make a nightly hotcopy of the repository which is backed up to
tape. The proverbial 'belt, braces, and a piece of string' backup approach.
This has run quite well for us for some years, through various 1.6.x
releases of subversion, and then 1.7.x releases.
I was recently away for a few weeks, and returned to find the pull sync
was failing, not able to lock the repository. I suspected some transient
problem, likely a network glitch, had corrupted the target. We were
running subversion 1.7.5 on both the source and target repositories.
I re-created the target repository, re-initialised the synchronisation,
and tried again. After 91 revisions copied, the svnsync command aborted.
I thought, maybe something about that revision? So I recreated the
repository, checked the permissions, and started again. After some 40000
revisions, the svnsync command aborted again.
I have since updated the target server to subversion 1.7.6, svnsync
still crashes, always at a different point.
I have just updated the source server to subversion 1.7.6, I have not
had the chance to try svnsync again, but I am not hopeful.
So, currently, we have
Production server, svnsync source repository, Ubuntu 10.04 with a hand
built subversion 1.7.6
Backup server, svnsync target repository, Ubuntu 12.04 with a hand built
subversion 1.7.6
The repository has around 77000 revisions currently.
The push svnsync still works correctly, but I am not sure if this is chance.
Unfortunately our mail gateway has a history of blocking some mails from
this list, so I will try to follow up via web archives. Replies cc'd to
me should also make it through (I hope).
Tony Butt
CEA Technologies
Canberra, Australia.
Re: svnsync fails on a 'pull' operation
Posted by Philip Martin <ph...@wandisco.com>.
Philip Martin <ph...@wandisco.com> writes:
> The code appears to assume that child_dirent is a longer string than
> parent_dirent.
This is now fixed on trunk and proposed for backport to 1.7. Thanks for
the report!
--
Philip
Re: svnsync fails on a 'pull' operation
Posted by Philip Martin <ph...@wandisco.com>.
Tony Butt <to...@cea.com.au> writes:
> On Tue, 2012-08-21 at 12:08 +0100, Daniel Shahaf wrote:
>> Daniel Shahaf wrote on Tue, Aug 21, 2012 at 12:01:28 +0100:
>> > You copy and paste the error message into the email for us.
>>
>> I meant to say: "You forgot to copy"
> I found the correct ulimit setting to enable core dumps, and was able to
> use that to obtain a stack trace - see attached.
>
> According to gdb, the failing function call is this memcmp(), called
> from svn_dirent_skip_ancestor().
>
> 1440 if (0 != memcmp(parent_dirent, child_dirent, len))
> 1441 return NULL; /* parent_dirent is no ancestor of child_dirent */
>
> The string arguments are evident from the stack trace.
> parent_dirent=0xb47faeb0 "/auspar/trunk/Modules/VirtualShip/Trunk/RadarGeometry/RadarGeometryManager/Trunk/Test/TestApplication/Build/Win32-x86-MULTI/DevelopmentShared.gpj"
> child_dirent=0xb468ef80 "/auspar/trunk/Modules/VirtualShip/Trunk/RadarGeometry/RadarGeometryManager/Trunk/Test/TestApplication/Build/Win32-x86-MULTI"
> len == 147
>
> Strangely enough, parent and child seem to back back to front. I am
> assuming the memcmp is running off the end of child_dirent, and causing
> my failure.
The code appears to assume that child_dirent is a longer string than
parent_dirent. It really should not be making that assumption.
--
Philip
Re: svnsync fails on a 'pull' operation
Posted by Tony Butt <to...@cea.com.au>.
On Tue, 2012-08-21 at 12:08 +0100, Daniel Shahaf wrote:
> Daniel Shahaf wrote on Tue, Aug 21, 2012 at 12:01:28 +0100:
> > You copy and paste the error message into the email for us.
>
> I meant to say: "You forgot to copy"
I found the correct ulimit setting to enable core dumps, and was able to
use that to obtain a stack trace - see attached.
According to gdb, the failing function call is this memcmp(), called
from svn_dirent_skip_ancestor().
1440 if (0 != memcmp(parent_dirent, child_dirent, len))
1441 return NULL; /* parent_dirent is no ancestor of child_dirent */
The string arguments are evident from the stack trace.
parent_dirent=0xb47faeb0 "/auspar/trunk/Modules/VirtualShip/Trunk/RadarGeometry/RadarGeometryManager/Trunk/Test/TestApplication/Build/Win32-x86-MULTI/DevelopmentShared.gpj"
child_dirent=0xb468ef80 "/auspar/trunk/Modules/VirtualShip/Trunk/RadarGeometry/RadarGeometryManager/Trunk/Test/TestApplication/Build/Win32-x86-MULTI"
len == 147
Strangely enough, parent and child seem to back back to front. I am
assuming the memcmp is running off the end of child_dirent, and causing
my failure.
This was built with gcc 4.6.3-1ubuntu5, using '-O2 -g'
I will try setting a breakpoint here, and looking for a consistent
pattern.
--
Tony Butt <to...@cea.com.au>
CEA Technologies
Re: svnsync fails on a 'pull' operation
Posted by Tony Butt <to...@cea.com.au>.
On Tue, 2012-08-21 at 12:08 +0100, Daniel Shahaf wrote:
> Daniel Shahaf wrote on Tue, Aug 21, 2012 at 12:01:28 +0100:
> > You copy and paste the error message into the email for us.
>
> I meant to say: "You forgot to copy"
OK - here is the failure with 1.7.6 server as source and destination,
which I ran last night. I didn't past it, as there is not much to see,
but that is sometimes an indication itself - sorry.
Copied properties for revision 1.
Committed revision 1.
...
Copied properties for revision 10612.
Committed revision 10613.
Copied properties for revision 10613.
Segmentation fault (core dumped)
The apache log at the source end shows this:
...
[21/Aug/2012:19:29:04 +1000] tjb rev-proplist r10612
[21/Aug/2012:19:29:05 +1000] tjb replay / r10612
[21/Aug/2012:19:29:05 +1000] tjb rev-proplist r10613
[21/Aug/2012:19:29:05 +1000] tjb replay / r10613
[21/Aug/2012:19:29:06 +1000] tjb rev-proplist r10614
[21/Aug/2012:19:29:06 +1000] tjb replay / r10614
And then nothing.
From previous events, this occurs with (apparently) random revisions,
never the same revision at the failure point.
--
Tony Butt <to...@cea.com.au>
CEA Technologies
Re: svnsync fails on a 'pull' operation
Posted by Tony Butt <to...@cea.com.au>.
On Thu, 2012-08-23 at 10:28 +1000, Tony Butt wrote:
> On Tue, 2012-08-21 at 12:08 +0100, Daniel Shahaf wrote:
> > Daniel Shahaf wrote on Tue, Aug 21, 2012 at 12:01:28 +0100:
> > > You copy and paste the error message into the email for us.
> >
> > I meant to say: "You forgot to copy"
> I found the correct ulimit setting to enable core dumps, and was able to
> use that to obtain a stack trace - see attached.
>
> According to gdb, the failing function call is this memcmp(), called
> from svn_dirent_skip_ancestor().
>
> 1440 if (0 != memcmp(parent_dirent, child_dirent, len))
> 1441 return NULL; /* parent_dirent is no ancestor of child_dirent */
>
> The string arguments are evident from the stack trace.
> parent_dirent=0xb47faeb0 "/auspar/trunk/Modules/VirtualShip/Trunk/RadarGeometry/RadarGeometryManager/Trunk/Test/TestApplication/Build/Win32-x86-MULTI/DevelopmentShared.gpj"
> child_dirent=0xb468ef80 "/auspar/trunk/Modules/VirtualShip/Trunk/RadarGeometry/RadarGeometryManager/Trunk/Test/TestApplication/Build/Win32-x86-MULTI"
> len == 147
>
> Strangely enough, parent and child seem to back back to front. I am
> assuming the memcmp is running off the end of child_dirent, and causing
> my failure.
> This was built with gcc 4.6.3-1ubuntu5, using '-O2 -g'
>
> I will try setting a breakpoint here, and looking for a consistent
> pattern.
>
OK, I set the breakpoint, and ran it.
In all cases, either the parent_dirent is identical to child_dirent, or
the parent_dirent is longer than the child_dirent !
I expect that the memcmp fails based on the vagaries of dynamic memory
allocation, and how much usable space is left at the end of the shorter
string.
I still don't know if this is unique to my build, or a broader problem.
--
Tony Butt <to...@cea.com.au>
CEA Technologies
Re: svnsync fails on a 'pull' operation
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Daniel Shahaf wrote on Tue, Aug 21, 2012 at 12:01:28 +0100:
> You copy and paste the error message into the email for us.
I meant to say: "You forgot to copy"
Re: svnsync fails on a 'pull' operation
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
You copy and paste the error message into the email for us.