You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Morten Mertner <mo...@mertner.com> on 2004/07/11 20:21:40 UTC
Subversion/BerkeleyDB lock problem (part II)
Hi Again,
I posted about a week ago regarding Subversion locking up the repository
and
needing recovery before coming back alive. I was advised to upgrade
BerkeleyDB,
and so I did (from 4.1.25-p1 to 4.2.52-p2).
Alas, it just happened again. Shutting down apache wasn't even enough
this time,
I had to reboot the server in order to make the "svnadmin recover"
command not
just hang (I waited about 5 mins and not hearing any disk activity or
seeing the
process being alive, I couldn't even terminate it using ctrl-c, so it
ate the ssh session
too). This has happened only once before and I thought it was a fluke,
but it's
definately related to the Subversion lockups.
Like I said in my last report, this is a real show-stopper. Having to
reboot the server
to keep the repository alive is just, well, you know.
I'm on a Gentoo Linux running a gentoo-patched 2.6.7 kernel, using
Subversion
1.0.4 with patches "subversion-db4" and "subversion-1.0.4-pre-1.0.5".
Packages are compiled for generic i686 using CFLAGS "-march=athlon-xp
-O2 -pipe
-fomit-frame-pointer" (unless overridden in the gentoo package ebuilds;
sometimes
the maintainers disable compile options to bypass options that would
break packages).
The problem has been present ever since 0.37 (when I first installed
Subversion),
although with increasing frequency in the last 2 months.
Any help in resolving this is highly appreciated!
Yours,
Morten
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion/BerkeleyDB lock problem (part II)
Posted by Morten Mertner <mo...@mertner.com>.
Hi All,
thanks for the suggestions, I'll use them the next time I have a hard lock.
Meanwhile, the repository has locked once more and I've created a zip file
before running svanadmin recover (no problems with this one this time).
http://www.mertner.com/mm/svn/svn-full.zip is a 330mb archive with the
full repository and all transaction logs.
http://www.mertner.com/mm/svn/svn-minimal.zip is a 94mb archive with
only the repository and the last few transaction logs.
http://www.mertner.com/mm/svn/svn-trlogs.zip is a 3mb archive with
the transaction logs from the previous hard lockup. Not sure if this is
useful to anyone, but at least the archive isn't very big ;)
Yours,
Morten
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion/BerkeleyDB lock problem (part II)
Posted by Ben Collins-Sussman <su...@collab.net>.
On Sun, 2004-07-11 at 20:56, Morten Mertner wrote:
> If it might be of any relevance the source repository stores no secrets
> (it's used for an
> open source project), so I can zip it up and put it online if that might
> be useful. I didn't
> think of this before now so actually doing it will have to wait until it
> happens again.
If it hapens again, I would also try using 'db_recover' for recovery
instead of 'svnadmin recover'. The former isn't waiting for locks on a
shared repository lockfile in locks/, while the latter is.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion/BerkeleyDB lock problem (part II)
Posted by Morten Mertner <mo...@mertner.com>.
Hi Ben,
>>I had to reboot the server in order to make the "svnadmin recover"
>>command not
>>just hang (I waited about 5 mins and not hearing any disk activity or
>>seeing the
>>process being alive, I couldn't even terminate it using ctrl-c, so it
>>ate the ssh session
>>too).
>>
>>
>
>Can you remind me of the details? It's not a shock that 'svnadmin
>recover' might not be interruptable by control-C... but couldn't you
>just open a new shell (ssh session, or local terminal, or whatever) and
>'kill -9' the process? Why did you have to reboot?
>
I could indeed open a new ssh session and kill the process, but without
being able to run
the svnadmin recover the repository would remain offline.
Once rebooted the recovery took about 5 seconds to complete.
If it might be of any relevance the source repository stores no secrets
(it's used for an
open source project), so I can zip it up and put it online if that might
be useful. I didn't
think of this before now so actually doing it will have to wait until it
happens again.
Yours,
Morten
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion/BerkeleyDB lock problem (part II)
Posted by Ben Collins-Sussman <su...@collab.net>.
On Sun, 2004-07-11 at 15:21, Morten Mertner wrote:
> I had to reboot the server in order to make the "svnadmin recover"
> command not
> just hang (I waited about 5 mins and not hearing any disk activity or
> seeing the
> process being alive, I couldn't even terminate it using ctrl-c, so it
> ate the ssh session
> too).
Can you remind me of the details? It's not a shock that 'svnadmin
recover' might not be interruptable by control-C... but couldn't you
just open a new shell (ssh session, or local terminal, or whatever) and
'kill -9' the process? Why did you have to reboot?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion/BerkeleyDB lock problem (part II)
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Sunday, July 11, 2004 10:21 PM +0200 Morten Mertner <mo...@mertner.com>
wrote:
> Like I said in my last report, this is a real show-stopper. Having to reboot
> the server
> to keep the repository alive is just, well, you know.
Unless there is a kernel problem related to shared memory (possible with
2.6.7, but I don't know), you shouldn't have to reboot. The one thing I'd
suggest doing is trying to compile bdb with debugging symbols enabled (i.e.
only '-g' and no '-O2') and try to get a backtrace of where svnadmin hangs.
-- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Subversion/BerkeleyDB lock problem (part II)
Posted by Mark Benedetto King <mb...@lowlatency.com>.
On Sun, Jul 11, 2004 at 10:21:40PM +0200, Morten Mertner wrote:
>
> Alas, it just happened again. Shutting down apache wasn't even enough
> this time,
> I had to reboot the server in order to make the "svnadmin recover"
> command not
> just hang (I waited about 5 mins and not hearing any disk activity or
> seeing the
> process being alive, I couldn't even terminate it using ctrl-c, so it
> ate the ssh session
> too). This has happened only once before and I thought it was a fluke,
> but it's
> definately related to the Subversion lockups.
svnadmin recover will block until all other locks on the repository
are released.
If you find this happening, try using "lsof" to determine which
other processes have the repository locked, and kill those ones
(while leaving the svnadmin recover running).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org