You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Morten Mertner <mo...@mertner.com> on 2004/07/03 22:05:48 UTC
Subversion repository breaks and needs recovery
Hi All,
About every few days and at least once a week Subversion has broken
something for itself and will only respond after svnadmin recover has
been executed. As this requires me to manually shutdown the Apache
server to prevent simultanous access, it has become a definite
show-stopper.
This issue has been present since 0.37 up to and including 1.0.5. I am
running Subversion on a Gentoo Linux box and the all repository access
occurs through WebDAV (Apache2 and mod_svn).
If there are any logging options or other mechanisms I can enable in
order to provide more details on the problem, or of I can otherwise be
helpful in troubleshooting this, let me know. It's a major problem for
me and I would really like to see it solved.
Yours,
Morten
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion repository breaks and needs recovery
Posted by Max Bowsher <ma...@ukf.net>.
Morten Mertner wrote:
> I believe you are right in that it is a locking problem, as the svnadmin
> process
> will lock/wait indefinately if I try to run the recovery without
> shutting down
> Apache.
That's not a locking *problem*. That is the svn libraries using locking to prevent you from destroying your repository by running
recovery whilst it is still in use.
There are several layers of lock.
1) Subversion repository locking.
2) BDB environment locking.
3) BDB data locking.
Here, you are observing level 1 functioning correctly.
When your repository is hanging, I believe the culprit is a stale lock at level 2.
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion repository breaks and needs recovery
Posted by Morten Mertner <mo...@mertner.com>.
Hi Max,
>It sounds like processes are exiting uncleanly whilst holding Berkeley DB shared region mutexes.
>
>Please tell us in detail what processes access the repository, and what operations are usually being attempted when the repository
>becomes wedged.
>
>Look out for any ways in which repository-accessing processes could be exiting abnormally.
>
>
The following is an excerpt from my Apache log and shows ALL hits made
on the
Subversion repository between two failures. It is the shortest period
I've found
between errors, and includes only 29 successful accesses:
[00:51:57] "REPORT /svn/repos/projects/gentle HTTP/1.1" 500 239
[01:29:03] "GET /svn/repos/projects/gentle/Source/ HTTP/1.1" 200 2103
[01:35:55] "GET /svn/repos/projects/gentle/Source/ HTTP/1.1" 200 2103
[01:40:57] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[01:40:58] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[01:40:58] "PROPFIND /svn/repos/projects/gentle HTTP/1.1" 207 708
[01:40:59] "PROPFIND /svn/repos/!svn/vcc/default HTTP/1.1" 207 457
[01:40:59] "REPORT /svn/repos/!svn/bc/445/projects/gentle HTTP/1.1" 200 472
[01:52:05] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[01:52:05] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[01:52:06] "PROPFIND /svn/repos/projects/gentle HTTP/1.1" 207 708
[01:52:06] "PROPFIND /svn/repos/!svn/vcc/default HTTP/1.1" 207 457
[01:52:07] "REPORT /svn/repos/!svn/bc/445/projects/gentle HTTP/1.1" 200 472
[02:11:00] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[02:11:01] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[02:11:01] "PROPFIND /svn/repos/projects/gentle HTTP/1.1" 207 708
[02:11:02] "PROPFIND /svn/repos/!svn/vcc/default HTTP/1.1" 207 457
[02:11:02] "REPORT /svn/repos/!svn/bc/445/projects/gentle HTTP/1.1" 200 472
[02:14:34] "GET /svn/repos/projects/gentle/Source/ HTTP/1.1" 200 2103
[02:21:02] "GET /svn/repos/projects/gentle/Source/ HTTP/1.1" 200 2103
[02:41:03] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[02:41:04] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[02:41:04] "PROPFIND /svn/repos/projects/gentle HTTP/1.1" 207 708
[02:41:04] "PROPFIND /svn/repos/!svn/vcc/default HTTP/1.1" 207 457
[02:41:05] "REPORT /svn/repos/!svn/bc/445/projects/gentle HTTP/1.1" 200 472
[02:52:22] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[02:52:22] "REPORT /svn/repos/projects/gentle HTTP/1.1" 200 147
[02:52:23] "PROPFIND /svn/repos/projects/gentle HTTP/1.1" 207 708
[02:52:23] "PROPFIND /svn/repos/!svn/vcc/default HTTP/1.1" 207 457
[02:52:24] "REPORT /svn/repos/!svn/bc/445/projects/gentle HTTP/1.1" 200 472
[03:11:06] "REPORT /svn/repos/projects/gentle HTTP/1.1" 500 239
Another thing to note is that none of the above should cause any changes to
the repository.
I believe you are right in that it is a locking problem, as the svnadmin
process
will lock/wait indefinately if I try to run the recovery without
shutting down
Apache.
Yours,
Morten
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion repository breaks and needs recovery
Posted by Max Bowsher <ma...@ukf.net>.
Morten Mertner wrote:
> Hi All,
>
> About every few days and at least once a week Subversion has broken
> something for itself and will only respond after svnadmin recover has
> been executed. As this requires me to manually shutdown the Apache
> server to prevent simultanous access, it has become a definite
> show-stopper.
>
> This issue has been present since 0.37 up to and including 1.0.5. I am
> running Subversion on a Gentoo Linux box and the all repository access
> occurs through WebDAV (Apache2 and mod_svn).
>
> If there are any logging options or other mechanisms I can enable in
> order to provide more details on the problem, or of I can otherwise be
> helpful in troubleshooting this, let me know. It's a major problem for
> me and I would really like to see it solved.
It sounds like processes are exiting uncleanly whilst holding Berkeley DB shared region mutexes.
Please tell us in detail what processes access the repository, and what operations are usually being attempted when the repository
becomes wedged.
Look out for any ways in which repository-accessing processes could be exiting abnormally.
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion repository breaks and needs recovery
Posted by Ben Reser <be...@reser.org>.
On Sun, Jul 04, 2004 at 12:05:48AM +0200, Morten Mertner wrote:
> About every few days and at least once a week Subversion has broken
> something for itself and will only respond after svnadmin recover has
> been executed. As this requires me to manually shutdown the Apache
> server to prevent simultanous access, it has become a definite
> show-stopper.
>
> This issue has been present since 0.37 up to and including 1.0.5. I am
> running Subversion on a Gentoo Linux box and the all repository access
> occurs through WebDAV (Apache2 and mod_svn).
>
> If there are any logging options or other mechanisms I can enable in
> order to provide more details on the problem, or of I can otherwise be
> helpful in troubleshooting this, let me know. It's a major problem for
> me and I would really like to see it solved.
Is the Apache server crashing? Can you enable core dumps for Apache and
rebuild subversion with --enable-maintainer-mode? Then use gdb to
provide us a backtrace of what happened?
Also it might be helpful if you can find the access_logs for the
operation/operations that broke the repo.
Beyond that hopefully one of the bdb experts can make some suggestions
that might help figure out how the repo is borken and that might give us
some clues...
--
Ben Reser <be...@reser.org>
http://ben.reser.org
"Conscience is the inner voice which warns us somebody may be looking."
- H.L. Mencken
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion repository breaks and needs recovery
Posted by Morten Mertner <mo...@mertner.com>.
Hi Max,
>>The one currently installed on the system is 4.1.25_p1-r3.
>>
>>
>Upgrade to 4.2 .
>Experience on these lists has shown that BDB has data corruption bugs,
>particularly, but not limited to, multiprocessor and
>hyperthreading systems.
>
>
I'm on a uniprocessor using 2.6 kernel (w/o nptl), but have upgraded to
4.2.52-p2. I'll let you know if it happens again - thanks for all your
help (and the locking clarification)!
Also, thanks for working on Subversion! I'm in the process of
convincing work to switch from VSS to Subversion, but have had some
qualms about it given the stability I've seen. I hope this will resolve
them so that it can be rolled out before VSS 2005 is released.
Yours,
Morten
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion repository breaks and needs recovery
Posted by Max Bowsher <ma...@ukf.net>.
Morten Mertner wrote:
> Hi,
>
>> What version of Berkeley DB is Subversion using?
>>
>>
> The one currently installed on the system is 4.1.25_p1-r3.
Upgrade to 4.2 .
Experience on these lists has shown that BDB has data corruption bugs, particularly, but not limited to, multiprocessor and
hyperthreading systems.
Max.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion repository breaks and needs recovery
Posted by Morten Mertner <mo...@mertner.com>.
Hi,
>What version of Berkeley DB is Subversion using?
>
>
The one currently installed on the system is 4.1.25_p1-r3.
Yours,
Morten
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion repository breaks and needs recovery
Posted by kf...@collab.net.
Morten Mertner <mo...@mertner.com> writes:
> About every few days and at least once a week Subversion has broken
> something for itself and will only respond after svnadmin recover has
> been executed. As this requires me to manually shutdown the Apache
> server to prevent simultanous access, it has become a definite
> show-stopper.
>
> This issue has been present since 0.37 up to and including 1.0.5. I am
> running Subversion on a Gentoo Linux box and the all repository access
> occurs through WebDAV (Apache2 and mod_svn).
>
> If there are any logging options or other mechanisms I can enable in
> order to provide more details on the problem, or of I can otherwise be
> helpful in troubleshooting this, let me know. It's a major problem for
> me and I would really like to see it solved.
What version of Berkeley DB is Subversion using?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org