Detecting graceful restart/stop in child processes

[Bill Moseley <mo...@hank.org>]

I'm running mod_perl/2.0.4 with a very large keep-alive set in my
httpd.conf.

Is there a way to detect when a graceful stop or restart has been requested
in the child apache processes?
I tried $SIG{USR1} and $SIG{WINCH} but I'm not seeing them catch the signal.

I'd like to be able to detect when a graceful stop or restart has been
requested so that I can add a Connection: close header to the response.


-- 
Bill Moseley
moseley@hank.org

Re: Detecting graceful restart/stop in child processes

[Bill Moseley <mo...@hank.org>]

On Mon, Dec 7, 2009 at 7:01 AM, Bill Moseley <mo...@hank.org>\
>
>
> So, it would be more convenient if a graceful restart could be done on just
> Apache and that would trigger closing the connections on the next response
> freeing up the process to exit.  That's the part I'm trying to get working.
>

This was pretty trivial once I remembered a little XS.

One interesting thing I realized (which makes sense if you think about it)
is that  MaxRequestsPerChild is not "requests" but connections.  IIRC, the
parent has to track the request count and it only knows about connections.
So, with persistent connections you don't really know the number of
requests.  And with memory leaks it's probably the requests that I'm
interested in.

I suppose I could count requests in the application process and call
child_terminate.  Or use Apache2::SizeLimit as that's really the reason to
limit number of requests.



-- 
Bill Moseley
moseley@hank.org

Re: Detecting graceful restart/stop in child processes

[Bill Moseley <mo...@hank.org>]

On Mon, Dec 7, 2009 at 1:18 AM, André Warnier <aw...@ice-sa.com> wrote:

> Bill Moseley wrote:
>
>>
>> Again, I've got keep alives set for a long time.
>>
>
> Well, isn't that your problem then ?
>
KeepAlive connections were introduced at a time when establishing and
> tearing down TCP connections were relatively expensive things to do.
> With modern servers however, this is less important.
>
> By setting KeepAliveTimeOut long, you essentially "block" a child (and all
> its resources), doing nothing but waiting for a follow-up request, for a
> long time after the last related request from the browser has come in, for
> the dubious benefit of avoiding the setup of a new connection.
>

This is true for my test environment I described, but that was just for
testing and was on purpose.

Under production a load balancer is used between the client and the
Apache/mod_perl servers.  It works as a proxy where the connection to the
client is separate from the connection to the pool of backend processes on
each server.  The balancer manages this pool of persistent connections ready
to handle requests.  Clients make separate persistent connections on the
front of the balancer, too.  But these are not directly connected to any one
process.  Requests come in from the presistent clients and the balancer
finds a free backend process to handle the request.  The backend services
the request very quickly and then is available to service other requests
while the client receives its response.  The keep-alive timeouts are
different on both sides of the balancer.

If the balancer maintains persistent connections to the back-end processes
then the problem is how to gracefully shut down a server.  Just doing a
graceful stop won't work because the connections keeps the process alive.
One way is to pull the server from the balancer  pool, wait for existing
requests to finish (only a few seconds) and then stop the server.   Problem
there is that it's not always the same people running the web servers and
balancers.

So, it would be more convenient if a graceful restart could be done on just
Apache and that would trigger closing the connections on the next response
freeing up the process to exit.  That's the part I'm trying to get working.




-- 
Bill Moseley
moseley@hank.org

Re: Detecting graceful restart/stop in child processes

[André Warnier <aw...@ice-sa.com>]

Bill Moseley wrote:
> 
> Again, I've got keep alives set for a long time.

Well, isn't that your problem then ?
KeepAlive connections were introduced at a time when establishing and 
tearing down TCP connections were relatively expensive things to do.
With modern servers however, this is less important.

By setting KeepAliveTimeOut long, you essentially "block" a child (and 
all its resources), doing nothing but waiting for a follow-up request, 
for a long time after the last related request from the browser has come 
in, for the dubious benefit of avoiding the setup of a new connection.

Re: Detecting graceful restart/stop in child processes

[Bill Moseley <mo...@hank.org>]

On Sun, Dec 6, 2009 at 9:51 AM, André Warnier <aw...@ice-sa.com> wrote:

>
> I am not the ultimate authority here, but I think you are going to have
> several problems with this idea.
> - the first one is that under Windows (where Apache also runs) there are no
> such signals.
> - the second one is that the signal, if signal there is, is probably
> addressed to the main Apache instance, not to the children processes.
> I am not sure that the Apache main process then tells the children to stop
> via a signal, or via some other mechanism.
> - assuming that a child process (or thread) has received (from the main
> Apache process) the order to terminate, it will do so after it finishes the
> current request, by which time it will be too late to add your header.
> But maybe I'm wrong with all of these.  Let's wait for a guru here to
> comment..
>

I'm not concerned with Windows.  I think all children in the process group
see the signals -- at least that's what I'm seeing.

Again, I've got keep alives set for a long time.   I can start up Apache and
make a request with firefox which establishes a connection with one
process.  Repeated request use the same connection (and process) due to the
keep alive. Then I do a graceful stop and just the parent process and that
one child process remain.

Netstat can also show that the connection is persistent to the child
process.

That process stays around serving requests until either the client (firefox)
closes the connection or a response sends a Connection: close header to the
client.  Then both the parent and child exit.  (I send a Connection: close
header when a query parameter is set.)

But, if I set a signal handler (either via $SIG or with a POSIX handler) I
can catch USR1 in the child process on a graceful stop, but it seems just
the act of catching the signal prevents the process from exiting.  That is,
the query parameter trick can still trigger closing the connection but the
process does not exit.

So, yes, an IPC and Apache guru would be the next step. ;)



-- 
Bill Moseley
moseley@hank.org

Re: Detecting graceful restart/stop in child processes

[André Warnier <aw...@ice-sa.com>]

Bill Moseley wrote:
> I'm running mod_perl/2.0.4 with a very large keep-alive set in my
> httpd.conf.
> 
> Is there a way to detect when a graceful stop or restart has been requested
> in the child apache processes?
> I tried $SIG{USR1} and $SIG{WINCH} but I'm not seeing them catch the signal.
> 
> I'd like to be able to detect when a graceful stop or restart has been
> requested so that I can add a Connection: close header to the response.
> 
> 
I am not the ultimate authority here, but I think you are going to have 
several problems with this idea.
- the first one is that under Windows (where Apache also runs) there are 
no such signals.
- the second one is that the signal, if signal there is, is probably 
addressed to the main Apache instance, not to the children processes.
I am not sure that the Apache main process then tells the children to 
stop via a signal, or via some other mechanism.
- assuming that a child process (or thread) has received (from the main 
Apache process) the order to terminate, it will do so after it finishes 
the current request, by which time it will be too late to add your header.
But maybe I'm wrong with all of these.  Let's wait for a guru here to 
comment..