You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Dave Rodgman <da...@arm.com> on 2005/11/04 16:56:43 UTC
svnadmin dump hangs
Hi,
I'm having issues with svnadmin dump, which occasionally hangs for no
apparent reason. I'm calling it from a post-commit hook for backup
purposes. I've verified that
- repeating the same command by hand completes successfully
- this is not due to disc-space issues
- this is happening fairly often - maybe once a week with around
5000 commits per week
- this is not related to the size of the commit - it's happened with
small and bigish commits
Server is v1.2.0 on fsfs backend. Should I raise a defect?
Suggestions gratefully received
Dave
Some data follows -
a recent occurrence:
hook script does:
svnadmin dump /svn/repositories/devsys/rvct -r 93301 --incremental
--deltas > /tmp/incremental_dump_r93301_d20051102171903000000
which hangs - ps auxf reveals:
httpd 31798 0.0 0.0 5764 1536 ? S Nov02
0:00 /usr/bin/perl /svn/repositories/devsys/rvct/hooks/post-commit /svn/repositories/devsys/rvct 93301
httpd 31802 0.0 0.0 2480 948 ? S Nov02 0:00 \_ sh -c
svnadmin dump /svn/repositories/devsys/rvct -r 93301 --incremental
--deltas > /tmp/incremental_dump_r93301_d2005110217190300httpd 31803
0.0 0.1 7836 3076 ? S Nov02 0:00 \_ svnadmin
dump /svn/repositories/devsys/rvct -r 93301 --incremental --deltas
The dump file is incomplete; repeating the command by hand (which
completes in about 6s) shows that it's about 25% complete.
This process is still running if anyone has any suggestions about how I
might debug it.
output of svnadmin --version:
#svnadmin --version
svnadmin, version 1.2.0 (r14790)
compiled May 31 2005, 16:58:10
=======================================================
from an older incident, where the commit is small enough to include the
dump file:
ps auxf:
httpd 29554 0.0 0.1 5092 3704 ? S Jun14 0:00
/usr/bin/perl /svn/repositories/devsys/sourcecontrol/hooks/post-commit
/svn/repositories/devsys/sourcecontrol 1956
httpd 29557 0.0 0.0 2476 1140 ? S Jun14 0:00 \_ sh -c
svnadmin dump /svn/repositories/devsys/sourcecontrol -r 1956
--incremental --deltas > /tmp/incremental_dump_r1956_d200506141
httpd 29558 0.0 0.1 6128 2380 ? S Jun14 0:00 \_
svnadmin dump /svn/repositories/devsys/sourcecontrol -r 1956
--incremental --deltas
httpd 7239 0.0 0.1 5088 3700 ? S Jun14 0:00
/usr/bin/perl /svn/repositories/devsys/sourcecontrol/hooks/post-commit
/svn/repositories/devsys/sourcecontrol 1959
httpd 7242 0.0 0.0 2472 1136 ? S Jun14 0:00 \_ sh -c
svnadmin dump /svn/repositories/devsys/sourcecontrol -r 1959
--incremental --deltas > /tmp/incremental_dump_r1959_d200506141
httpd 7243 0.0 0.1 6124 2376 ? S Jun14 0:00 \_
svnadmin dump /svn/repositories/devsys/sourcecontrol -r 1959
--incremental --deltas
but doing e.g. svnadmin dump /svn/repositories/devsys/sourcecontrol -r
1959 --incremental --deltas works fine.
Contents of e.g. /tmp/incremental_dump_r1959_d200506141 are incomplete
(this is not a disc space problem in /tmp, btw):
SVN-fs-dump-format-version: 3
UUID: daa62ece-fce0-0310-acf6-c38c6792efcd
Revision-number: 1959
Prop-content-length: 210
Content-length: 210
K 7
svn:log
V 106
Greatly improved error messages; correct TeamAssist behaviour for some
corner cases with multiple products K 10 svn:author V 8 drodgman K 8
svn:date V 27 2005-06-14T11:31:49.624675Z PROPS-END
Node-path: hooks/tags/stable-20050614
Node-kind: dir
Node-action: add
Node-copyfrom-rev: 1892
Node-copyfrom-path: hooks/trunk
Node-path: hooks/tags/stable-20050614/commit
Node-kind: file
Node-action: delete
Node-path: hooks/tags/stable-20050614/commit
Node-kind: file
Node-action: add
Node-copyfrom-rev: 1891
Node-copyfrom-path: hooks/trunk/commit
Node-path: hooks/tags/stable-20050614/post-commit.d
Node-kind: dir
Node-action: delete
Node-path: hooks/tags/stable-20050614/post-commit.d
Node-kind: dir
Node-action: add
Node-copyfrom-rev: 1891
Node-copyfrom-path: hooks/trunk/post-commit.d
Node-path: hooks/tags/stable-20050614/post-revprop-change.d
Node-kind: dir
Node-action: delete
Node-path: hooks/tags/stable-20050614/post-revprop-change.d
Node-kind: dir
Node-action: add
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnadmin dump hangs
Posted by Erik Huelsmann <eh...@gmail.com>.
On 11/4/05, Dave Rodgman <da...@arm.com> wrote:
> Hi,
>
> I'm having issues with svnadmin dump, which occasionally hangs for no
> apparent reason. I'm calling it from a post-commit hook for backup
> purposes. I've verified that
>
> - repeating the same command by hand completes successfully
> - this is not due to disc-space issues
> - this is happening fairly often - maybe once a week with around
> 5000 commits per week
> - this is not related to the size of the commit - it's happened with
> small and bigish commits
>
> Server is v1.2.0 on fsfs backend. Should I raise a defect?
>
> Suggestions gratefully received
>
> Dave
>
> Some data follows -
>
> a recent occurrence:
>
> hook script does:
>
> svnadmin dump /svn/repositories/devsys/rvct -r 93301 --incremental
> --deltas > /tmp/incremental_dump_r93301_d20051102171903000000
>
What happens if you redirect stderr to /dev/null? It may be that the
pipe is getting filled. I vaguely remember we have forgotten to read
the stderr pipe after spawning the post-commit process. Which means
you can - maybe (and depending on the system) - write 4k output to it
before it blocks.
bye,
Erik.
Re: svnadmin dump hangs
Posted by Philip Martin <ph...@codematters.co.uk>.
kfogel@collab.net writes:
> Hmmm. I think this is worth an issue, yes (thanks for the detailed
> report). I wish we had a 100% reliable reproduction recipe, but this
> is probably as good a start as we could ask for with this kind of bug.
>
> -Karl
>
> Dave Rodgman <da...@arm.com> writes:
>> I'm having issues with svnadmin dump, which occasionally hangs for no
>> apparent reason. I'm calling it from a post-commit hook for backup
>> purposes. I've verified that
>>
>> - repeating the same command by hand completes successfully
>> - this is not due to disc-space issues
>> - this is happening fairly often - maybe once a week with around
>> 5000 commits per week
>> - this is not related to the size of the commit - it's happened with
>> small and bigish commits
>>
>> Server is v1.2.0 on fsfs backend. Should I raise a defect?
1.2.0 is a bit out-of-date, in particular it has the fsfs multi-thread
lock bug. If you are using a threaded httpd or svnserve then it may
be worth while upgrading to 1.2.3. I don't know if it's the cause of
your problem, the bug showed up as a race if multiple commits occurred
at the same time and it's possible that it would cause problems for
the post-commit scripts.
>>
>> Suggestions gratefully received
>>
>> Dave
>>
>> Some data follows -
>>
>> a recent occurrence:
>>
>> hook script does:
>>
>> svnadmin dump /svn/repositories/devsys/rvct -r 93301 --incremental
>> --deltas > /tmp/incremental_dump_r93301_d20051102171903000000
>>
>> which hangs - ps auxf reveals:
>>
>> httpd 31798 0.0 0.0 5764 1536 ? S Nov02
>> 0:00 /usr/bin/perl /svn/repositories/devsys/rvct/hooks/post-commit /svn/repositories/devsys/rvct 93301
>> httpd 31802 0.0 0.0 2480 948 ? S Nov02 0:00 \_ sh -c
>> svnadmin dump /svn/repositories/devsys/rvct -r 93301 --incremental
>> --deltas > /tmp/incremental_dump_r93301_d2005110217190300httpd 31803
>> 0.0 0.1 7836 3076 ? S Nov02 0:00 \_ svnadmin
>> dump /svn/repositories/devsys/rvct -r 93301 --incremental --deltas
>>
>> The dump file is incomplete; repeating the command by hand (which
>> completes in about 6s) shows that it's about 25% complete.
>>
>> This process is still running if anyone has any suggestions about how I
>> might debug it.
Attach a debugger and get a stack trace. On Linux show us what gdb's
bt command displays:
$ gdb -p <NNN>
(gdb) bt
(gdb) q
$
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnadmin dump hangs
Posted by Martin Furter <mf...@rola.ch>.
Hello
I also dump the commited revisions in the hook script and never saw that
problem. My machine runs Solaris 9, SVN 1.2.1 and ra_dav.
An important difference might be that i run the dump script in background,
the post-commit hook contains only these lines:
trap "" 1
/srv/svn/scripts/post-commit-bg.sh "$1" "$2" < /dev/null > /dev/null 2>&1 &
The script post-commit-bg.sh then dumps the revision and sends commit
mails.
Martin
On Mon, 7 Nov 2005 kfogel@collab.net wrote:
> Hmmm. I think this is worth an issue, yes (thanks for the detailed
> report). I wish we had a 100% reliable reproduction recipe, but this
> is probably as good a start as we could ask for with this kind of bug.
>
> -Karl
>
> Dave Rodgman <da...@arm.com> writes:
>> I'm having issues with svnadmin dump, which occasionally hangs for no
>> apparent reason. I'm calling it from a post-commit hook for backup
>> purposes. I've verified that
>>
>> - repeating the same command by hand completes successfully
>> - this is not due to disc-space issues
>> - this is happening fairly often - maybe once a week with around
>> 5000 commits per week
>> - this is not related to the size of the commit - it's happened with
>> small and bigish commits
>>
>> Server is v1.2.0 on fsfs backend. Should I raise a defect?
>>
>> Suggestions gratefully received
>>
>> Dave
>>
>> Some data follows -
[...removed...]
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: svnadmin dump hangs
Posted by kf...@collab.net.
Hmmm. I think this is worth an issue, yes (thanks for the detailed
report). I wish we had a 100% reliable reproduction recipe, but this
is probably as good a start as we could ask for with this kind of bug.
-Karl
Dave Rodgman <da...@arm.com> writes:
> I'm having issues with svnadmin dump, which occasionally hangs for no
> apparent reason. I'm calling it from a post-commit hook for backup
> purposes. I've verified that
>
> - repeating the same command by hand completes successfully
> - this is not due to disc-space issues
> - this is happening fairly often - maybe once a week with around
> 5000 commits per week
> - this is not related to the size of the commit - it's happened with
> small and bigish commits
>
> Server is v1.2.0 on fsfs backend. Should I raise a defect?
>
> Suggestions gratefully received
>
> Dave
>
> Some data follows -
>
> a recent occurrence:
>
> hook script does:
>
> svnadmin dump /svn/repositories/devsys/rvct -r 93301 --incremental
> --deltas > /tmp/incremental_dump_r93301_d20051102171903000000
>
> which hangs - ps auxf reveals:
>
> httpd 31798 0.0 0.0 5764 1536 ? S Nov02
> 0:00 /usr/bin/perl /svn/repositories/devsys/rvct/hooks/post-commit /svn/repositories/devsys/rvct 93301
> httpd 31802 0.0 0.0 2480 948 ? S Nov02 0:00 \_ sh -c
> svnadmin dump /svn/repositories/devsys/rvct -r 93301 --incremental
> --deltas > /tmp/incremental_dump_r93301_d2005110217190300httpd 31803
> 0.0 0.1 7836 3076 ? S Nov02 0:00 \_ svnadmin
> dump /svn/repositories/devsys/rvct -r 93301 --incremental --deltas
>
> The dump file is incomplete; repeating the command by hand (which
> completes in about 6s) shows that it's about 25% complete.
>
> This process is still running if anyone has any suggestions about how I
> might debug it.
>
> output of svnadmin --version:
>
> #svnadmin --version
> svnadmin, version 1.2.0 (r14790)
> compiled May 31 2005, 16:58:10
>
> =======================================================
>
> from an older incident, where the commit is small enough to include the
> dump file:
>
> ps auxf:
>
> httpd 29554 0.0 0.1 5092 3704 ? S Jun14 0:00
> /usr/bin/perl /svn/repositories/devsys/sourcecontrol/hooks/post-commit
> /svn/repositories/devsys/sourcecontrol 1956
> httpd 29557 0.0 0.0 2476 1140 ? S Jun14 0:00 \_ sh -c
> svnadmin dump /svn/repositories/devsys/sourcecontrol -r 1956
> --incremental --deltas > /tmp/incremental_dump_r1956_d200506141
> httpd 29558 0.0 0.1 6128 2380 ? S Jun14 0:00 \_
> svnadmin dump /svn/repositories/devsys/sourcecontrol -r 1956
> --incremental --deltas
> httpd 7239 0.0 0.1 5088 3700 ? S Jun14 0:00
> /usr/bin/perl /svn/repositories/devsys/sourcecontrol/hooks/post-commit
> /svn/repositories/devsys/sourcecontrol 1959
> httpd 7242 0.0 0.0 2472 1136 ? S Jun14 0:00 \_ sh -c
> svnadmin dump /svn/repositories/devsys/sourcecontrol -r 1959
> --incremental --deltas > /tmp/incremental_dump_r1959_d200506141
> httpd 7243 0.0 0.1 6124 2376 ? S Jun14 0:00 \_
> svnadmin dump /svn/repositories/devsys/sourcecontrol -r 1959
> --incremental --deltas
>
> but doing e.g. svnadmin dump /svn/repositories/devsys/sourcecontrol -r
> 1959 --incremental --deltas works fine.
>
> Contents of e.g. /tmp/incremental_dump_r1959_d200506141 are incomplete
> (this is not a disc space problem in /tmp, btw):
>
> SVN-fs-dump-format-version: 3
>
> UUID: daa62ece-fce0-0310-acf6-c38c6792efcd
>
> Revision-number: 1959
> Prop-content-length: 210
> Content-length: 210
>
> K 7
> svn:log
> V 106
> Greatly improved error messages; correct TeamAssist behaviour for some
> corner cases with multiple products K 10 svn:author V 8 drodgman K 8
> svn:date V 27 2005-06-14T11:31:49.624675Z PROPS-END
>
> Node-path: hooks/tags/stable-20050614
> Node-kind: dir
> Node-action: add
> Node-copyfrom-rev: 1892
> Node-copyfrom-path: hooks/trunk
>
>
> Node-path: hooks/tags/stable-20050614/commit
> Node-kind: file
> Node-action: delete
>
> Node-path: hooks/tags/stable-20050614/commit
> Node-kind: file
> Node-action: add
> Node-copyfrom-rev: 1891
> Node-copyfrom-path: hooks/trunk/commit
>
>
>
>
> Node-path: hooks/tags/stable-20050614/post-commit.d
> Node-kind: dir
> Node-action: delete
>
> Node-path: hooks/tags/stable-20050614/post-commit.d
> Node-kind: dir
> Node-action: add
> Node-copyfrom-rev: 1891
> Node-copyfrom-path: hooks/trunk/post-commit.d
>
>
>
>
> Node-path: hooks/tags/stable-20050614/post-revprop-change.d
> Node-kind: dir
> Node-action: delete
>
> Node-path: hooks/tags/stable-20050614/post-revprop-change.d
> Node-kind: dir
> Node-action: add
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
--
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org