svn.collab.net hangage.

["C. Michael Pilato" <cm...@collab.net>]

(Hangage.  That word looks funny.  Heh.)

So, tonight I got email from our nightly backup script, notifying me
that one of the svn.collab.net repositories needed recovery.  Didn't
take long to figure out that it was the 'svn' repository.  Bum thing
was that 'svnadmin recover' failed -- said I needed to run recovery.
db_recover said the same thing.

Being a man with little free time and sleepy eyelids, I skipped the
diagnostics and went straight to our onsite backup.  We keep four
nights' worth of 'svnadmin hotcopy's for each hosted repository, plus
a nightly offsite backup.  The most recent backup (tonight's, of
r9759) was hosed in the same way that the live site was.  So I went to
last night's (r9743).  It was all good.  

(Very) fortunately, no log files had been cleaned from the live
repository since last night's backup.  I was able to 'svnadmin
recover' and 'svnadmin list-unused-dblogs | rm' it.  I then copied
over all the logfiles in the live site that were newer than the one
remaining used logfile in the backup (including the finished version
of that logfile), and did a 'db_recover -c' catastrophic recovery.  It
went without a hitch, the repository claimed to have 9760 revisions,
and 'svnadmin verify' checked out all good.  Problem solv--er, painted
over.

I wasn't really able to determine the cause of the DB_RECOVER loop.
The most informative message I got in the entire process was the one
about region errors begin detected.  I can't help but wonder if
perhaps the disk in that machine (which recently took a bumpy ride on
a rolling cart from CollabNet's old office across the street to our
new one) needs some TLC.  But I also can't admit to having a strong
conviction in that as a likely cause.

At any rate, things are purring again it seems, and I need some sleep.

(NOTE to fellow administrators: the busted repository has been
preserved as ~/svn.b0rked, and Apache logfiles have been copied into
~/svn.b0rked/logfiles for future examination).

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn.collab.net hangage.

[Jani Averbach <ja...@jaa.iki.fi>]

On 2004-05-17 23:25-0500, C. Michael Pilato wrote:

> I can't help but wonder if perhaps the disk in that machine (which
> recently took a bumpy ride on a rolling cart from CollabNet's old
> office across the street to our new one) needs some TLC.  But I also
> can't admit to having a strong conviction in that as a likely cause.

Might want try <http://smartmontools.sourceforge.net/> ?

BR, Jani

-- 
Jani Averbach 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn.collab.net hangage.

["C. Michael Pilato" <cm...@collab.net>]

Karl Chen <qu...@nospam.quarl.org> writes:

> >>>>> "C" == C Michael Pilato <cm...@collab.net> writes:
>     C> (NOTE to fellow administrators: the busted repository has
>     C> been preserved as ~/svn.b0rked, and Apache logfiles have
>     C> been copied into ~/svn.b0rked/logfiles for future
>     C> examination).
> 
> Is it possible to replay the actions to try to reb0rk it?

Not likely.  It'd be extremely hard to simulate an environment like
that, with multiple Apache processes all hitting the repository at
precise times, etc.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: svn.collab.net hangage.

[Karl Chen <qu...@nospam.quarl.org>]

>>>>> "C" == C Michael Pilato <cm...@collab.net> writes:
    C> (NOTE to fellow administrators: the busted repository has
    C> been preserved as ~/svn.b0rked, and Apache logfiles have
    C> been copied into ~/svn.b0rked/logfiles for future
    C> examination).

Is it possible to replay the actions to try to reb0rk it?

-- 
Karl 2004-05-17 23:28


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org