proxy health checks [was: Proxying OPTIONS *]

[Rainer Jung <ra...@kippdata.de>]

Jim Jagielski wrote:
> On Oct 2, 2007, at 5:56 PM, Ruediger Pluem wrote:
>> Slightly off topic, but this gives me the idea that we could use
>> OPTIONS * as some kind of ping / health check for pooled connections
>> in mod_proxy_http before sending a request (at least in the reverse
>> proxy case before sending a request that is not idempotent or after
>> the connection has not been used for a certain amount of time).
>> The current is_connected check has a race condition if the keepalive
>> timer of the backend server kills the connection just after our check
>> and before it received our request.
> 
> :)
> 
> We already do similar with AJP where, right after 
> ap_proxy_connect_backend()
> it does a quick check. I think we even discussed having HTTP do something
> similar awhile ago...

It would be nice to have the health check running asynchronously from 
the normal request handling (error detection for idle connections, 
before requests fail). Even more important, the recovery test could be 
done independently from request handling.

For this one would need some scheduling service (independent threads, 
API extension for callbacks). A simple start could be something like the 
monitor() hook. Although that one depends on the type of MPM and the 
module scheduling service should not be decoupled from the timing of 
important internal procedures.

Regards,

Rainer

Re: proxy health checks [was: Proxying OPTIONS *]

[Nick Kew <ni...@webthing.com>]

On Wed, 03 Oct 2007 15:04:52 +0200
Rainer Jung <ra...@kippdata.de> wrote:

> It would be nice to have the health check running asynchronously from 
> the normal request handling (error detection for idle connections, 
> before requests fail). Even more important, the recovery test could
> be done independently from request handling.

Indeed.  That could apply to other backends (such as mod_dbd) too.

> For this one would need some scheduling service (independent threads, 
> API extension for callbacks). A simple start could be something like
> the monitor() hook. Although that one depends on the type of MPM and
> the module scheduling service should not be decoupled from the timing
> of important internal procedures.

Patches welcome!

Maybe a scheduled event thread could be a standard feature for httpd-3,
to make this kind of thing straightforward.

-- 
Nick Kew

Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/

Re: proxy health checks [was: Proxying OPTIONS *]

["Akins, Brian" <Br...@turner.com>]

In "our" proxy, we launch an external helper app the does active health
checking of the origin servers.  This is a HEAD request on a configurable
(per origin "pool") uri (ie, http://host:port/url/blah).  When an origin
passes/fails a given number of checks it is marked up/down.  For example,
when an origin passes 2 health checks in a row, it is marked up; when it
fails 3 in a row, it is marked down.  We check each backend every x seconds
(x usually equal to 5).

The proxy module and the health checker are linked via a simple array in
shared memory.  The health checker marks the index of each origin as 1 (up)
or 0 (down).  

I have been contemplating writing a balancer module that implements this,
but haven't had the time.

With this method, we don't have to check every time, as we assume if the
origin is up in the health checker that it is up.

This works several million times a day for us :)

(Also, we do "true" connection pooling to the origins, but that's another
story...)


-- 
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies