minotaur down

[Nick Kew <ni...@webthing.com>]

minotaur has been down for six hour now, perhaps longer.
Given how many critical things live there, this is a problem.

From discussion on #asfinfra (IRC), it seems (correct me if
I'm wrong) that it can be accessed remotely by sufficiently
privileged users even when down, but there are very few
people with that privilege.

Shouldn't there be an emergency backup plan, including
a standby server that can be switched to for the public
pages, and enough people, distributed over different
timezones, equipped and privileged to switch to it?

-- 
Nick Kew

Re: minotaur down

[Joshua Slive <jo...@slive.ca>]

Nick Kew wrote:
> minotaur has been down for six hour now, perhaps longer.
> Given how many critical things live there, this is a problem.
> 
> From discussion on #asfinfra (IRC), it seems (correct me if
> I'm wrong) that it can be accessed remotely by sufficiently
> privileged users even when down, but there are very few
> people with that privilege.
> 
> Shouldn't there be an emergency backup plan, including
> a standby server that can be switched to for the public
> pages, and enough people, distributed over different
> timezones, equipped and privileged to switch to it?

Ooo... And fairy dust.  I think there should be fairy dust. ;-)

See the extended discussion about hiring paid infra help and feel free 
to come over to infrastructure@apache.org and start writing down a 
disaster recovery plan.

By the way, there is a live backup at httpd.eu.apache.org (and similarly 
for most other sites), but in order to make the switch in DNS we'd need 
access to minotaur.  Oops.

Joshua.