Recommendations for a "hot standby" subversion server

[Kylo Ginsberg <ky...@gmail.com>]

We have a half-dozen svn repositories hosted on a Fedora 7 box, the
largest of which is ~10GB/100k revisions.  Right now, our failsafe
strategy is nightly hotcopy's backed up to tape.  If the server were
to crash irrecoverably, we would have to bring up another pc, install
software as needed, restore the previous night's hotcopy, and go, all
of which could take a number of hours.  Thus, we'd like to explore a
"hot standby" server.  (Really it's just "warm": manual intervention
is ok, but it would be nice if it was relatively quick.)

My (very hazy) initial thought is to use 1.5 write-through proxying to
have a master server and eventually multiple slave servers (1 per
geographic region).  One slave would be designated as the backup
primary.  Since the hooks are different for a slave vs a master, each
server would have a set of standard hooks and a set of hooks we'd
switch to in the event the master went down.  Likewise, for apache
configurations.  If the master goes down, we switch hooks/configs on
all slaves.  If a slave goes down, we merely update dns to redirect to
another slave or perhaps to the master.

So the first questions are: does this make sense?  Is there a better
way?  Any gotchas I should consider?

In addition some questions/concerns re write-through proxying:
* svnbook mentions the concern about a failed svnsync leaving a slave
out-of-date until the next commit and mentions monitoring slaves
out-of-band to narrow this window.  However, revprop edits seem like a
bigger hole still: if a "synsync copy-revprops" fails for some reason
(e.g., slave down) what's the recommended practice?
* If a slave is down, will svnsync's to that slave (in the master's
post-commit hook) respond gracefully?
* svnbook seems to recommend forking off svnsync's, which allows for
the possibility of rapid back-to-back commits resulting in multiple
simultaneous svnsync's to the same slave.  Same question: is this
handled gracefully?

Thanks for any comments.
Kylo

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@subversion.tigris.org
For additional commands, e-mail: users-help@subversion.tigris.org

Re: Recommendations for a "hot standby" subversion server

[communitybot <co...@googlemail.com>]

>
> If the server were
> to crash irrecoverably, we would have to bring up another pc, install
> software as needed, restore the previous night's hotcopy, and go, all
> of which could take a number of hours.


There's your biggest problem - you've got to set up a new server in a panic!
Our Subversion server setup is on a virtual machine, so we simply keep a
copy of the VM and use backup/restore scripts that use hotcopies, which are
kept on other machines and tapes. And, of course, clear instructions on what
to do if the worst happens.

Having a 'lukewarm' (!) failsafe like that keeps your live setup simple and
allows you to quickly restore from anything between catastrophic hardware
failure right through to botched upgrades or repository corruption. Also
helps to avoid said botched upgrades by doing proper testing first.

On the negative side, you lose everything since your last hotcopy...