You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Bill Stoddard <bi...@wstoddard.com> on 2003/02/04 17:53:48 UTC
Graceful shutdown in 2.0
Has anyone ever thought about the best way to implement graceful
shutdown (perhaps with a timeout) in the server? This would be a
useful function to have where there is a cluster of http servers (in a
DMZ for instance) proxying requests to a farm backend application
servers. If you need to take an http server completely offline, you
could issue something like apachectl shutdown_graceful [timeout] and
prevent that server from accepting any new connections but allow the
existing requests to be completed w/o disruption (subject to a timeout).
Bill
Re: Graceful shutdown in 2.0
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
At 07:29 AM 2/5/2003, Hans Zaunere wrote:
>> David Burry wrote:
>>
>> > um, but if you're talking about shutting down the proxy itself (i.e. the
>> > whole service, cutting off all load balanced machines behind it) that's
>> > hardly graceful to begin with.... so why bother to make it graceful...
>
>I've wondered about this as well. Even for just single servers, it'd be nice
>to shutdown the server "gracefully". Often, I'm more comfortable with
>stopping and then starting the server than doing a reload, so a graceful
>shutdown would be useful, unless I'm missing something that already exists.
With the new interdependencies with DAV content... e.g. live files are
being modified while the server is operating, it seems like a forced shutdown
of a DAV server could cause havoc. Correct me if I'm off base here.
So perhaps graceful shutdown would be a good thing, perhaps even
ultra-graceful (allowing open keep alive connections to finish serving
all requests.) But at minimum we don't want to always cut off POST
or PUT requests mid-stream.
Bill
Re: Graceful shutdown in 2.0
Posted by David Burry <db...@tagnet.org>.
On our systems we just rename that "alteoncheck.txt" file to
"alteoncheck_DOWN.txt" when we're going to bring a server down (causing a
404 error for the health check, which stops all new requests), it
effectively does the same thing you describe without the hassle of writing a
handler. And yes it is very nice in that it's easily automated...
So, yes, it would be very nice to have a graceful shutdown, but it's not
necessarily high priority for those who have some sort of load balancer box
(not round robin DNS ;) because there are other relatively simple ways to
achieve the same effect...
Dave
----- Original Message -----
From: "Andrew Ho" <an...@tellme.com>
> On a more load balancer specific note, Alteons (and some other load
> balancers) use the concept of a health check URL. Our Alteons are
> configured for example to check for a specific URL (for example, the
> Alteon might do a "GET alteoncheck.txt HTTP/1.0" every 2 seconds).
>
> I had a plan originally to write a handler that accepts requests for this
> heartbeat check... on some signal (a particular request? an OS signal?) it
> would start returning an error for the heartbeat check case, but keep
> servicing all other requests as normal. Eventually, the Alteon would
> decide that that machine was bad, and the number of connections would fall
> to zero; it would then be safe to take the server out of rotation.
>
> The benefit of this scenario is that you don't have to touch the load
> balancer at all to get individual machines in and out of the load
> balancer. Also, this type of scenario is far more automatable (rather than
> telnetting into, say, a load balancer console interface and navigating
> menus, ugh).
Re: Graceful shutdown in 2.0
Posted by Andrew Ho <an...@tellme.com>.
Hello,
GL>If your proxy machines were being load balanced by an Alteon switch (or
GL>whatever) you could probably tell the switch to stop sending new
GL>connections to the proxy in question, while allowing the old connections
GL>to finish. Of course this is based on whether the Alteon (or whatever)
GL>config will let you do this of course.
For the Alteons on our production networks, we do exactly this. It works
pretty well.
GL>On the proxy side, maybe some generic function in Apache that is not
GL>proxy specific could signal Apache to go into a "disabled" state, where
GL>new connections get "connection refused", while old connections run
GL>their course. The load balancer I assume would detect this state and
GL>pass any new requests on to other machines. You could use ./apachectl
GL>disable and ./apachectl enable to achieve this.
This is exactly the graceful shutdown case that was suggested, which I am
wholeheartedly supportive of. (Cutting off requests that are in service
is, in my opinion, never good; graceful shutdown seems like a really
useful feature to have.)
On a more load balancer specific note, Alteons (and some other load
balancers) use the concept of a health check URL. Our Alteons are
configured for example to check for a specific URL (for example, the
Alteon might do a "GET alteoncheck.txt HTTP/1.0" every 2 seconds).
I had a plan originally to write a handler that accepts requests for this
heartbeat check... on some signal (a particular request? an OS signal?) it
would start returning an error for the heartbeat check case, but keep
servicing all other requests as normal. Eventually, the Alteon would
decide that that machine was bad, and the number of connections would fall
to zero; it would then be safe to take the server out of rotation.
The benefit of this scenario is that you don't have to touch the load
balancer at all to get individual machines in and out of the load
balancer. Also, this type of scenario is far more automatable (rather than
telnetting into, say, a load balancer console interface and navigating
menus, ugh).
Humbly,
Andrew
----------------------------------------------------------------------
Andrew Ho http://www.tellme.com/ andrew@tellme.com
Engineer info@tellme.com Voice 650-930-9062
Tellme Networks, Inc. 1-800-555-TELL Fax 650-930-9101
----------------------------------------------------------------------
Re: Graceful shutdown in 2.0
Posted by Hans Zaunere <za...@yahoo.com>.
--- Graham Leggett <mi...@sharp.fm> wrote:
> David Burry wrote:
>
> > um, but if you're talking about shutting down the proxy itself (i.e. the
> > whole service, cutting off all load balanced machines behind it) that's
> > hardly graceful to begin with.... so why bother to make it graceful...
>
> If your proxy machines were being load balanced by an Alteon switch (or
> whatever) you could probably tell the switch to stop sending new
> connections to the proxy in question, while allowing the old connections
> to finish. Of course this is based on whether the Alteon (or whatever)
> config will let you do this of course.
>
> On the proxy side, maybe some generic function in Apache that is not
> proxy specific could signal Apache to go into a "disabled" state, where
> new connections get "connection refused", while old connections run
> their course. The load balancer I assume would detect this state and
> pass any new requests on to other machines. You could use ./apachectl
> disable and ./apachectl enable to achieve this.
I've wondered about this as well. Even for just single servers, it'd be nice
to shutdown the server "gracefully". Often, I'm more comfortable with
stopping and then starting the server than doing a reload, so a graceful
shutdown would be useful, unless I'm missing something that already exists.
Hans
Re: Graceful shutdown in 2.0
Posted by Graham Leggett <mi...@sharp.fm>.
David Burry wrote:
> um, but if you're talking about shutting down the proxy itself (i.e. the
> whole service, cutting off all load balanced machines behind it) that's
> hardly graceful to begin with.... so why bother to make it graceful...
If your proxy machines were being load balanced by an Alteon switch (or
whatever) you could probably tell the switch to stop sending new
connections to the proxy in question, while allowing the old connections
to finish. Of course this is based on whether the Alteon (or whatever)
config will let you do this of course.
On the proxy side, maybe some generic function in Apache that is not
proxy specific could signal Apache to go into a "disabled" state, where
new connections get "connection refused", while old connections run
their course. The load balancer I assume would detect this state and
pass any new requests on to other machines. You could use ./apachectl
disable and ./apachectl enable to achieve this.
Regards,
Graham
--
-----------------------------------------
minfrin@sharp.fm "There's a moon
over Bourbon Street
tonight..."
Re: Graceful shutdown in 2.0
Posted by David Burry <db...@tagnet.org>.
um, but if you're talking about shutting down the proxy itself (i.e. the
whole service, cutting off all load balanced machines behind it) that's
hardly graceful to begin with.... so why bother to make it graceful...
I assumed you meant just gracefully shutting down one single load balanced
machine behind the proxy machine... you can do that already now by a)
configuring the proxy machine to stop routing (new) requests to it, b)
graceful restart the proxy machine to make the new config go into effect, c)
wait till the existing connections to that behind-the-proxy machine are
finished with a timeout if necessary (sort of part of the graceful restart
process in the proxy machine), then d) shut down the machine behind the
proxy, in that order. External users should not notice anything at all in
this scenario.
----- Original Message -----
From: "Bill Stoddard" <bi...@wstoddard.com>
To: <de...@httpd.apache.org>
Sent: Tuesday, February 04, 2003 12:25 PM
Subject: Re: Graceful shutdown in 2.0
> David Burry wrote:
> > The same effect is already possible by configuring your proxying machine
to
> > stop forwarding new requests to that box first....
>
> Yep, that's the idea. In the scenario I'm interested in, Apache httpd
> -is- the proxy machine!
>
> Bill
>
>
>
Re: Graceful shutdown in 2.0
Posted by Bill Stoddard <bi...@wstoddard.com>.
David Burry wrote:
> The same effect is already possible by configuring your proxying machine to
> stop forwarding new requests to that box first....
Yep, that's the idea. In the scenario I'm interested in, Apache httpd
-is- the proxy machine!
Bill
Re: Graceful shutdown in 2.0
Posted by David Burry <db...@tagnet.org>.
The same effect is already possible by configuring your proxying machine to
stop forwarding new requests to that box first.... Of course, it's possible
that different people manage the proxying service vs the back end apache
services, so I can see how it can be desireable to have this feature in
apache too, but still those two people should always be working pretty
closely together anyway...
Dave
----- Original Message -----
From: "Bill Stoddard" <bi...@wstoddard.com>
To: <de...@httpd.apache.org>
Sent: Tuesday, February 04, 2003 8:53 AM
Subject: Graceful shutdown in 2.0
> Has anyone ever thought about the best way to implement graceful
> shutdown (perhaps with a timeout) in the server? This would be a
> useful function to have where there is a cluster of http servers (in a
> DMZ for instance) proxying requests to a farm backend application
> servers. If you need to take an http server completely offline, you
> could issue something like apachectl shutdown_graceful [timeout] and
> prevent that server from accepting any new connections but allow the
> existing requests to be completed w/o disruption (subject to a timeout).
>
> Bill
>