Re: Work in progress: mod_proxy Health Check module

[Jim Jagielski <ji...@jaguNET.com>]

I am thinking about the actual health-check impl a bit more,
hence the lack of commits. As noted, initially I was thinking
about optional functions, defined in mod_proxy_http, mod_proxy_ajp,
etc. Then I started thinking that maybe doing it via providers
might be better, allowing for more "custom" health checks via
the end user. Then I started thinking about doing it rough-and-
ready and just adding 'em in mod_proxy_hcheck...

Before I spend too much time on this, any prefs one way or
another regarding this?

tia

Re: Work in progress: mod_proxy Health Check module

[Stefan Eissing <st...@greenbytes.de>]

> Am 09.01.2016 um 19:18 schrieb Jim Jagielski <ji...@jagunet.com>:
> 
> Here is the general concept:
> 
> The health check itself is done via mod_watchdog, and uses
> its callback impl. Each worker will have its own check interval,
> which can be changed via the new balancer member directives
> (hcinterval=, hcfails=,...); these will also be changable via
> the balancer manager.
> 
> There will be a set of health checks available, depending on
> the scheme of the backend. There will be:
> 
>  None
>  TCP: a simple 'is socket up check'
>  OPTIONS: for http only
>  HEAD: http only, sends HEAD /...
>  GET: http only, sends GET /...
>  CPING: ajp cping/cpong

For completeness: HTTP/2 has a PING frame.

And, yes, I might make an attempt at a http2 proxy modding sometime soon... >:-)

-Stefan

Re: Work in progress: mod_proxy Health Check module

[Jim Jagielski <ji...@jaguNET.com>]

What's currently in trunk is mostly ready for a more
in-depth look-see. Right now, only the tcp check is
complete, with others coming shortly.

Will work on the docs over the weekend.

Re: Work in progress: mod_proxy Health Check module

[Jim Jagielski <ji...@jaguNET.com>]

Here is the general concept:

The health check itself is done via mod_watchdog, and uses
its callback impl. Each worker will have its own check interval,
which can be changed via the new balancer member directives
(hcinterval=, hcfails=,...); these will also be changable via
the balancer manager.

There will be a set of health checks available, depending on
the scheme of the backend. There will be:

  None
  TCP: a simple 'is socket up check'
  OPTIONS: for http only
  HEAD: http only, sends HEAD /...
  GET: http only, sends GET /...
  CPING: ajp cping/cpong

For the checks that send a URI, we allow for a specific
check URI (hcuri) that gets appended to the worker's url.
We will also read in the response, headers and all, and
create a dummy request_rec from it. Why? That way we can
create 'conditions' which use ap_expr to check for things
like REQUEST_STATUS, etc. These conditions will be
basically stored ap_expr expressions (ala <If>) that will
be applied to the returned response from the backend. Thus,
we can check not only that the backend is "up" (ie, it
responded), but also check the response itself.

These would be one-off requests, connecting directly to
the backend via ap_proxy_acquire_connection()/
ap_proxy_connect_backend()/ap_proxy_connection_create() with
the *dummy* generic reverse proxy worker (instead of the
actual backend worker itself, if you grok). It may make sense
to in addition to having the generic forward and reverse proxy
workers, also have a hcheck one as well, specialized for
this purpose.

I like the idea of parallel requests, and could likely impl
that as a simple thread pool in the watchdog process itself,
but that might be v1.1 or v1.2; I want to get the basic flow
itself right, esp the logic checking. So v1.0 will be serial. :/

> On Jan 8, 2016, at 9:45 PM, Daniel Ruggeri <DR...@primary.net> wrote:
> 
> ++1 on this vein of thought. Implementing the health checking similar to
> the modularized lb methods makes the most sense to me. I can already
> think of several methods (file, conf, external script, lua script) of
> obtaining data on how to probe and react.
> 
> Just some thoughts worth considering... since you asked :-)
> *The module should not probe if a probe is already running against that
> backend (hang or takes too long to respond)
> *The probes to different backends should be done in parallel rather than
> serially to avoid pileups due to a slow responder
> *How will the existing balancer re-enable logic interfere with ERR backends?
>   *Maybe another flag that avoids retry if the monitor marked it down
> requiring the monitor to bring it back up before any traffic is allowed
> to go to it?
> 
> I've followed along with what you described by using watchdog to trigger
> the probing, but I'm curious where your thoughts are for submitting a
> request to the backend. Are you planning to simulate one from your
> module by directly connecting to the backend or is the plan to run the
> request through the existing proxy infrastructure (the former allows
> lots of control while the latter permits other directives to come into
> play that might be handy like disabling based on status code). I haven't
> seen the code, but your previous email said you were thinking of the
> former case.
> 
> P.S.
> Thanks for taking this on. It's been on my own todo list for a long time.
> 
> -- 
> Daniel Ruggeri
> 
> On 1/8/2016 1:09 PM, Jim Jagielski wrote:
>> Thx! Any other comments/suggestions?
>> 
>>> On Jan 8, 2016, at 10:53 AM, Eric Covener <co...@gmail.com> wrote:
>>> 
>>> On Fri, Jan 8, 2016 at 10:18 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>>>> I am thinking about the actual health-check impl a bit more,
>>>> hence the lack of commits. As noted, initially I was thinking
>>>> about optional functions, defined in mod_proxy_http, mod_proxy_ajp,
>>>> etc. Then I started thinking that maybe doing it via providers
>>>> might be better, allowing for more "custom" health checks via
>>>> the end user. Then I started thinking about doing it rough-and-
>>>> ready and just adding 'em in mod_proxy_hcheck...
>>>> 
>>>> Before I spend too much time on this, any prefs one way or
>>>> another regarding this?
>>> I haven't looked at the architecture so far -- but I think basics in
>>> mod_proxy_hcheck
>>> and at least one "standalone" provider (as an example) would be a good
>>> middle ground.
>>> 
>>> Maybe the standalone one would try to deal with the body whereas the
>>> built-in ones
>>> would be TCP up, SSL up, status-code, etc.
> 

Re: Work in progress: mod_proxy Health Check module

[Jim Jagielski <ji...@jaguNET.com>]

> On Jan 8, 2016, at 9:45 PM, Daniel Ruggeri <DR...@primary.net> wrote:
> 
> *The probes to different backends should be done in parallel rather than
> serially to avoid pileups due to a slow responder

I just implemented use of a threadpool to accommodate that. ;)

Re: Work in progress: mod_proxy Health Check module

[Daniel Ruggeri <DR...@primary.net>]

++1 on this vein of thought. Implementing the health checking similar to
the modularized lb methods makes the most sense to me. I can already
think of several methods (file, conf, external script, lua script) of
obtaining data on how to probe and react.

Just some thoughts worth considering... since you asked :-)
*The module should not probe if a probe is already running against that
backend (hang or takes too long to respond)
*The probes to different backends should be done in parallel rather than
serially to avoid pileups due to a slow responder
*How will the existing balancer re-enable logic interfere with ERR backends?
   *Maybe another flag that avoids retry if the monitor marked it down
requiring the monitor to bring it back up before any traffic is allowed
to go to it?

I've followed along with what you described by using watchdog to trigger
the probing, but I'm curious where your thoughts are for submitting a
request to the backend. Are you planning to simulate one from your
module by directly connecting to the backend or is the plan to run the
request through the existing proxy infrastructure (the former allows
lots of control while the latter permits other directives to come into
play that might be handy like disabling based on status code). I haven't
seen the code, but your previous email said you were thinking of the
former case.

P.S.
Thanks for taking this on. It's been on my own todo list for a long time.

-- 
Daniel Ruggeri

On 1/8/2016 1:09 PM, Jim Jagielski wrote:
> Thx! Any other comments/suggestions?
>
>> On Jan 8, 2016, at 10:53 AM, Eric Covener <co...@gmail.com> wrote:
>>
>> On Fri, Jan 8, 2016 at 10:18 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>>> I am thinking about the actual health-check impl a bit more,
>>> hence the lack of commits. As noted, initially I was thinking
>>> about optional functions, defined in mod_proxy_http, mod_proxy_ajp,
>>> etc. Then I started thinking that maybe doing it via providers
>>> might be better, allowing for more "custom" health checks via
>>> the end user. Then I started thinking about doing it rough-and-
>>> ready and just adding 'em in mod_proxy_hcheck...
>>>
>>> Before I spend too much time on this, any prefs one way or
>>> another regarding this?
>> I haven't looked at the architecture so far -- but I think basics in
>> mod_proxy_hcheck
>> and at least one "standalone" provider (as an example) would be a good
>> middle ground.
>>
>> Maybe the standalone one would try to deal with the body whereas the
>> built-in ones
>> would be TCP up, SSL up, status-code, etc.

Re: Work in progress: mod_proxy Health Check module

[Jim Jagielski <ji...@jaguNET.com>]

Thx! Any other comments/suggestions?

> On Jan 8, 2016, at 10:53 AM, Eric Covener <co...@gmail.com> wrote:
> 
> On Fri, Jan 8, 2016 at 10:18 AM, Jim Jagielski <ji...@jagunet.com> wrote:
>> I am thinking about the actual health-check impl a bit more,
>> hence the lack of commits. As noted, initially I was thinking
>> about optional functions, defined in mod_proxy_http, mod_proxy_ajp,
>> etc. Then I started thinking that maybe doing it via providers
>> might be better, allowing for more "custom" health checks via
>> the end user. Then I started thinking about doing it rough-and-
>> ready and just adding 'em in mod_proxy_hcheck...
>> 
>> Before I spend too much time on this, any prefs one way or
>> another regarding this?
> 
> I haven't looked at the architecture so far -- but I think basics in
> mod_proxy_hcheck
> and at least one "standalone" provider (as an example) would be a good
> middle ground.
> 
> Maybe the standalone one would try to deal with the body whereas the
> built-in ones
> would be TCP up, SSL up, status-code, etc.

Re: Work in progress: mod_proxy Health Check module

[Eric Covener <co...@gmail.com>]

On Fri, Jan 8, 2016 at 10:18 AM, Jim Jagielski <ji...@jagunet.com> wrote:
> I am thinking about the actual health-check impl a bit more,
> hence the lack of commits. As noted, initially I was thinking
> about optional functions, defined in mod_proxy_http, mod_proxy_ajp,
> etc. Then I started thinking that maybe doing it via providers
> might be better, allowing for more "custom" health checks via
> the end user. Then I started thinking about doing it rough-and-
> ready and just adding 'em in mod_proxy_hcheck...
>
> Before I spend too much time on this, any prefs one way or
> another regarding this?

I haven't looked at the architecture so far -- but I think basics in
mod_proxy_hcheck
and at least one "standalone" provider (as an example) would be a good
middle ground.

Maybe the standalone one would try to deal with the body whereas the
built-in ones
would be TCP up, SSL up, status-code, etc.