You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Karl Fogel <kf...@newton.ch.collab.net> on 2002/01/25 00:10:24 UTC
refrobnicated repository again
The DB got hung again. I did the usual routine, with one minor
variation: at Greg Stein's suggestion, I shut down Apache before
running recover. Recovery took less than a minute this time:
[svn@svn db]$ ps aux | grep apach
root 20235 0.0 0.7 4352 1976 ? S 14:32 0:00 /usr/local/apache
svn 20236 0.9 8.6 24972 22020 ? S 14:32 1:47 /usr/local/apache
svn 20237 0.0 6.5 20320 16628 ? S 14:32 0:11 /usr/local/apache
svn 20238 1.5 17.9 49052 45792 ? S 14:32 3:06 /usr/local/apache
svn 20239 0.0 2.7 9444 7104 ? S 14:32 0:02 /usr/local/apache
svn 20240 0.3 5.7 16976 14760 ? S 14:32 0:37 /usr/local/apache
svn 20242 0.0 1.5 6488 3984 ? S 14:32 0:00 /usr/local/apache
svn 20254 0.0 2.3 8344 5900 ? S 14:35 0:02 /usr/local/apache
svn 20267 0.2 16.0 45872 41036 ? S 14:39 0:26 /usr/local/apache
svn 20274 0.0 2.8 9552 7268 ? S 14:41 0:01 /usr/local/apache
svn 20290 0.1 6.0 18664 15372 ? S 14:46 0:21 /usr/local/apache
svn 20291 0.1 11.3 32540 29032 ? S 14:46 0:16 /usr/local/apache
svn 21510 0.0 1.0 5220 2784 ? S 17:42 0:00 /usr/local/apache
svn 21511 0.0 0.7 4412 2036 ? S 17:43 0:00 /usr/local/apache
svn 21512 0.0 0.7 4412 2036 ? S 17:44 0:00 /usr/local/apache
svn 21513 0.0 0.7 4412 2036 ? S 17:47 0:00 /usr/local/apache
svn 21514 0.0 0.7 4412 2036 ? S 17:47 0:00 /usr/local/apache
svn 21515 0.0 0.7 4412 2036 ? S 17:47 0:00 /usr/local/apache
[svn@svn db]$ su
[root@svn db]# /usr/local/apache2/bin/apachectl stop
/usr/local/apache2/bin/apachectl stop: httpd stopped
[root@svn db]# exit
[svn@svn db]$ ls
__db.001 __db.003 __db.005 nodes revisions transactions
__db.002 __db.004 log.0000000135 representations strings
[svn@svn db]$ /usr/local/BerkeleyDB.4.0/bin/db_recover -ve
db_recover: Finding last valid log LSN: file: 135 offset 8145058
db_recover: Checkpoint at: [135][8144597]
db_recover: Checkpoint LSN: [135][8139917]
db_recover: Previous checkpoint: [135][7880280]
db_recover: Checkpoint at: [135][7880280]
db_recover: Checkpoint LSN: [135][7880280]
db_recover: Previous checkpoint: [135][7469824]
db_recover: Recovery starting from [135][7880280]
db_recover: Recovery complete at Thu Jan 24 17:49:40 2002
db_recover: Maximum transaction ID 80045ee9 Recovery checkpoint [135][8145519]
db_recover: Recovery complete at Thu Jan 24 17:49:40 2002
db_recover: Maximum transaction id 80000000 Recovery checkpoint [135][8145519]
[svn@svn db]$ su
[root@svn db]# /usr/local/apache2/bin/apachectl start
/usr/local/apache2/bin/apachectl start: httpd started
[root@svn db]# exit
[svn@svn db]$
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: refrobnicated repository again
Posted by Sander Striker <st...@apache.org>.
Hi,
It was a message with these properties:
Subject: Re: SVN UPDATE ERROR:
Message-ID: <87...@codematters.co.uk>
if that helps.
Sander
> -----Original Message-----
> From: Justin Erenkrantz [mailto:jerenkrantz@ebuilt.com]
> Sent: 25 January 2002 20:44
> To: dev@subversion.tigris.org
> Subject: Re: refrobnicated repository again
>
>
> On Fri, Jan 25, 2002 at 10:44:36AM -0800, Greg Stein wrote:
> > Philip posted a program which can cause the same hang in a development
> > environment. *That* is where we'd be capturing stacktraces and stuff.
> > svn.collab.net is a production box, so we shouldn't be mucking around with
> > it trying to capture traces.
>
> You mentioned that before, but I obviously missed that post. Can
> someone point out the program? -- justin
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Fri, Jan 25, 2002 at 10:44:36AM -0800, Greg Stein wrote:
> Philip posted a program which can cause the same hang in a development
> environment. *That* is where we'd be capturing stacktraces and stuff.
> svn.collab.net is a production box, so we shouldn't be mucking around with
> it trying to capture traces.
You mentioned that before, but I obviously missed that post. Can
someone point out the program? -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Greg Stein <gs...@lyra.org>.
On Fri, Jan 25, 2002 at 08:14:41AM -0800, Justin Erenkrantz wrote:
> On Fri, Jan 25, 2002 at 04:10:47PM +0000, Philip Martin wrote:
> > gdb will do it. Just use 'gdb /path/to/executable PID'.
>
> Right, but I was thinking of something that the admin could slap into
> a file easily. I doubt that we'd have the time to debug it while it
> is hung. You want to capture the bt and then restart the server. The
> backtrace can then be sent to dev@svn to pounce on. -- justin
Philip posted a program which can cause the same hang in a development
environment. *That* is where we'd be capturing stacktraces and stuff.
svn.collab.net is a production box, so we shouldn't be mucking around with
it trying to capture traces.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Fri, Jan 25, 2002 at 04:10:47PM +0000, Philip Martin wrote:
> gdb will do it. Just use 'gdb /path/to/executable PID'.
Right, but I was thinking of something that the admin could slap into
a file easily. I doubt that we'd have the time to debug it while it
is hung. You want to capture the bt and then restart the server. The
backtrace can then be sent to dev@svn to pounce on. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Philip Martin <ph...@codematters.co.uk>.
Justin Erenkrantz <je...@ebuilt.com> writes:
> On Fri, Jan 25, 2002 at 10:11:53AM -0600, Karl Fogel wrote:
> > Justin Erenkrantz <je...@ebuilt.com> writes:
> > > I don't know what OS you are running the server on. (Oh, I bet
> > > FreeBSD from the ps options.)
> >
> > RedHat Linux, actually:
> >
> > $ uname -a
> > Linux svn.collab.net 2.4.2-2 #1 Sun Apr 8 20:41:30 EDT 2001 i686 unknown
>
> Ah, I was afraid of that. That pstack version is only for FreeBSD.
> *sigh* The only version of pstack I can find for Linux is busted.
> Perhaps attaching to gdb is our only hope then to get a callstack.
>
> Anyone else have any ideas? -- justin
gdb will do it. Just use 'gdb /path/to/executable PID'.
--
Philip
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Fri, Jan 25, 2002 at 10:11:53AM -0600, Karl Fogel wrote:
> Justin Erenkrantz <je...@ebuilt.com> writes:
> > I don't know what OS you are running the server on. (Oh, I bet
> > FreeBSD from the ps options.)
>
> RedHat Linux, actually:
>
> $ uname -a
> Linux svn.collab.net 2.4.2-2 #1 Sun Apr 8 20:41:30 EDT 2001 i686 unknown
Ah, I was afraid of that. That pstack version is only for FreeBSD.
*sigh* The only version of pstack I can find for Linux is busted.
Perhaps attaching to gdb is our only hope then to get a callstack.
Anyone else have any ideas? -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Justin Erenkrantz <je...@ebuilt.com> writes:
> I don't know what OS you are running the server on. (Oh, I bet
> FreeBSD from the ps options.)
RedHat Linux, actually:
$ uname -a
Linux svn.collab.net 2.4.2-2 #1 Sun Apr 8 20:41:30 EDT 2001 i686 unknown
> Perhaps pstack would be useful to find out where those httpd
> processes are:
>
> http://www.apache.org/~jerenkrantz/pstack.tar.gz
>
> If you aren't familiar with pstack, if you have the -g flag on
> compilation (not sure if it requires this or not), it'll print
> the call stack for all threads/processes. We obviously need
> a back trace. =) -- justin
Awesome! Thanks, that may help...
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Thu, Jan 24, 2002 at 06:10:24PM -0600, Karl Fogel wrote:
> The DB got hung again. I did the usual routine, with one minor
> variation: at Greg Stein's suggestion, I shut down Apache before
> running recover. Recovery took less than a minute this time:
I don't know what OS you are running the server on. (Oh, I bet
FreeBSD from the ps options.)
Perhaps pstack would be useful to find out where those httpd
processes are:
http://www.apache.org/~jerenkrantz/pstack.tar.gz
If you aren't familiar with pstack, if you have the -g flag on
compilation (not sure if it requires this or not), it'll print
the call stack for all threads/processes. We obviously need
a back trace. =) -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
cmpilato@collab.net writes:
> Not really. We could do something that small inline in the
> post-commit hook, right?
>
> for logfile in `db_archive -a -h ${repos}/db`; do rm ${logfile}; done
>
> I mean, that's all there is to it, correct?
Do we really want this overhead on every commit? How about a cron job?
Ah, bikeshed. Happy with either way. :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Branko Čibej <br...@xbc.nu>.
Eric M. Hopper wrote:
>On Fri, Jan 25, 2002 at 12:55:56PM -0600, Ben Collins-Sussman wrote:
>
>>"Eric M. Hopper" <ho...@omnifarious.org> writes:
>>
>>>>$ /usr/local/BerkeleyDB.4.0/bin/db_archive -a -h ${repos}/db | xargs rm
>>>>
>>>>Much nicer than a loop...
>>>>
>>> Someone should create a directory in the repository who's name
>>>has a space in it, which contains other directories and files with
>>>spaces, double quotes, single quotes, backticks and carraige returns in
>>>their names. This will save you many headaches later.
>>>
>>Eric, the db_archive program spits out the names of DB logfiles that
>>can be safely deleted. It has nothing to do with names of things
>>stored in the virtual fs. :-)
>>
>
> I suspected the db_archive program might produce shell
>metacharacter clean ouput. Some things just jump out at me when reading
>code, and I figured making that general comment was a good idea.
>
> I do believe, for example, that the svn:ignore property is
>stored in a form that will not work for files with certain characters in
>their filenames.
>
That's right. If you have a file with a newline in the name, and want to
put it's name in svn:ignore, you're scewed.
>*grin*,
>
ditto, but /my/ grin is evil! :-)
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
"Eric M. Hopper" <ho...@omnifarious.org> writes:
> I suspected the db_archive program might produce shell
> metacharacter clean ouput. Some things just jump out at me when reading
> code, and I figured making that general comment was a good idea.
>
> I do believe, for example, that the svn:ignore property is
> stored in a form that will not work for files with certain characters in
> their filenames.
Yes, these are conscious decisions, Eric.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by "Eric M. Hopper" <ho...@omnifarious.org>.
On Fri, Jan 25, 2002 at 12:55:56PM -0600, Ben Collins-Sussman wrote:
> "Eric M. Hopper" <ho...@omnifarious.org> writes:
>
> > > $ /usr/local/BerkeleyDB.4.0/bin/db_archive -a -h ${repos}/db | xargs rm
> > >
> > > Much nicer than a loop...
> >
> > Someone should create a directory in the repository who's name
> > has a space in it, which contains other directories and files with
> > spaces, double quotes, single quotes, backticks and carraige returns in
> > their names. This will save you many headaches later.
>
> Eric, the db_archive program spits out the names of DB logfiles that
> can be safely deleted. It has nothing to do with names of things
> stored in the virtual fs. :-)
I suspected the db_archive program might produce shell
metacharacter clean ouput. Some things just jump out at me when reading
code, and I figured making that general comment was a good idea.
I do believe, for example, that the svn:ignore property is
stored in a form that will not work for files with certain characters in
their filenames.
*grin*,
--
"It does me no injury for my neighbor to say there are twenty gods or no God.
It neither picks my pocket nor breaks my leg." --- Thomas Jefferson
"Go to Heaven for the climate, Hell for the company." -- Mark Twain
-- Eric Hopper (hopper@omnifarious.org http://www.omnifarious.org/~hopper) --
Re: refrobnicated repository again
Posted by Ben Collins-Sussman <su...@collab.net>.
"Eric M. Hopper" <ho...@omnifarious.org> writes:
> > $ /usr/local/BerkeleyDB.4.0/bin/db_archive -a -h ${repos}/db | xargs rm
> >
> > Much nicer than a loop...
>
> Someone should create a directory in the repository who's name
> has a space in it, which contains other directories and files with
> spaces, double quotes, single quotes, backticks and carraige returns in
> their names. This will save you many headaches later.
Eric, the db_archive program spits out the names of DB logfiles that
can be safely deleted. It has nothing to do with names of things
stored in the virtual fs. :-)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by "Eric M. Hopper" <ho...@omnifarious.org>.
On Fri, Jan 25, 2002 at 10:40:28AM -0800, Greg Stein wrote:
> On Fri, Jan 25, 2002 at 10:36:02AM -0600, cmpilato@collab.net wrote:
> >...
> > Not really. We could do something that small inline in the
> > post-commit hook, right?
> >
> > for logfile in `db_archive -a -h ${repos}/db`; do rm ${logfile}; done
> >
> > I mean, that's all there is to it, correct?
>
> Invoke it from somewhere, somehow. Cron, foreground, background...
>
> But simpler:
>
> $ /usr/local/BerkeleyDB.4.0/bin/db_archive -a -h ${repos}/db | xargs rm
>
> Much nicer than a loop...
Someone should create a directory in the repository who's name
has a space in it, which contains other directories and files with
spaces, double quotes, single quotes, backticks and carraige returns in
their names. This will save you many headaches later.
*grin*,
--
"It does me no injury for my neighbor to say there are twenty gods or no God.
It neither picks my pocket nor breaks my leg." --- Thomas Jefferson
"Go to Heaven for the climate, Hell for the company." -- Mark Twain
-- Eric Hopper (hopper@omnifarious.org http://www.omnifarious.org/~hopper) --
Re: refrobnicated repository again
Posted by Greg Stein <gs...@lyra.org>.
On Fri, Jan 25, 2002 at 10:36:02AM -0600, cmpilato@collab.net wrote:
>...
> Not really. We could do something that small inline in the
> post-commit hook, right?
>
> for logfile in `db_archive -a -h ${repos}/db`; do rm ${logfile}; done
>
> I mean, that's all there is to it, correct?
Invoke it from somewhere, somehow. Cron, foreground, background...
But simpler:
$ /usr/local/BerkeleyDB.4.0/bin/db_archive -a -h ${repos}/db | xargs rm
Much nicer than a loop...
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by cm...@collab.net.
Ben Collins-Sussman <su...@collab.net> writes:
> But we need a *separate* commit-hook script that cleans up logfiles in
> the live repository., don't you think?
Not really. We could do something that small inline in the
post-commit hook, right?
for logfile in `db_archive -a -h ${repos}/db`; do rm ${logfile}; done
I mean, that's all there is to it, correct?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Ben Collins-Sussman <su...@collab.net>.
cmpilato@collab.net writes:
> Ben Collins-Sussman <su...@collab.net> writes:
>
> > I suspect the '15 minutes' thing was happening because there were
> > waaaaay too many useless log files in the repository... and db_recover
> > was needlessly running over them.
> >
> > (The 'hot-backup.py' script that we run on each commit wasn't fully
> > doing it's job: it was removing useless logs from the *backup* copy,
> > not the live repository. I think this still needs to be fixed!)
>
> Heh...I was just making this change. Of course, the Berkeley docs say
> that we should NOT be deleting those logfiles, but be archiving them
> to removable media instead for catastrophic recovery.
>
hot-backup.py should do one thing: make a backup. It does that. If
it wants to remove useless log files in the backup, that's fine with
me. If we want to be extra-paranoid and keep them around, that's ok,
I'm -0. (We've already got multiple backups lying around.)
But we need a *separate* commit-hook script that cleans up logfiles in
the live repository., don't you think?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Greg Stein <gs...@lyra.org>.
On Fri, Jan 25, 2002 at 10:23:05AM -0600, cmpilato@collab.net wrote:
>..
> Heh...I was just making this change. Of course, the Berkeley docs say
> that we should NOT be deleting those logfiles, but be archiving them
> to removable media instead for catastrophic recovery.
>
> Thoughts?
I believe it says to keep them so that you can completely replay the log
from scratch, and regenerate your database.
Presuming you keep backup copies of the database, I don't see why that would
be necessary.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by cm...@collab.net.
Ben Collins-Sussman <su...@collab.net> writes:
> I suspect the '15 minutes' thing was happening because there were
> waaaaay too many useless log files in the repository... and db_recover
> was needlessly running over them.
>
> (The 'hot-backup.py' script that we run on each commit wasn't fully
> doing it's job: it was removing useless logs from the *backup* copy,
> not the live repository. I think this still needs to be fixed!)
Heh...I was just making this change. Of course, the Berkeley docs say
that we should NOT be deleting those logfiles, but be archiving them
to removable media instead for catastrophic recovery.
Thoughts?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Greg Stein <gs...@lyra.org>.
On Fri, Jan 25, 2002 at 10:21:55AM -0600, Ben Collins-Sussman wrote:
>...
> (The 'hot-backup.py' script that we run on each commit wasn't fully
> doing it's job: it was removing useless logs from the *backup* copy,
> not the live repository. I think this still needs to be fixed!)
hot_backup should not be changing the repository; certainly not deleting the
log files in there.
I believe it wasn't removing the log files from the backups because there
was a stale /usr/backup/lock file sitting around. You probably want to put a
try/finally block in there to ensure the lock file is removed.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Ben Collins-Sussman <su...@collab.net>.
Garrett Rooney <ro...@electricjellyfish.net> writes:
> On Thu, Jan 24, 2002 at 06:10:24PM -0600, Karl Fogel wrote:
> > The DB got hung again. I did the usual routine, with one minor
> > variation: at Greg Stein's suggestion, I shut down Apache before
> > running recover. Recovery took less than a minute this time:
>
> well, that's encouraging anyway...
>
> the 'it used to take 3 seconds and now it takes 15 minutes' thing was
> kind of freaking me out...
I suspect the '15 minutes' thing was happening because there were
waaaaay too many useless log files in the repository... and db_recover
was needlessly running over them.
(The 'hot-backup.py' script that we run on each commit wasn't fully
doing it's job: it was removing useless logs from the *backup* copy,
not the live repository. I think this still needs to be fixed!)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: refrobnicated repository again
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On Thu, Jan 24, 2002 at 06:10:24PM -0600, Karl Fogel wrote:
> The DB got hung again. I did the usual routine, with one minor
> variation: at Greg Stein's suggestion, I shut down Apache before
> running recover. Recovery took less than a minute this time:
well, that's encouraging anyway...
the 'it used to take 3 seconds and now it takes 15 minutes' thing was
kind of freaking me out...
-garrett
--
garrett rooney Unix was not designed to stop you from
rooneg@electricjellyfish.net doing stupid things, because that would
http://electricjellyfish.net/ stop you from doing clever things.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org