httpd-proxy-scoreboard

[Jean-frederic Clere <jf...@gmail.com>]

Hi,

I have started to write a "generic" health-checker for mod_proxy. I 
would like to change the macro PROXY_WORKER_IS_USABLE() to a routine in 
proxy_util.c.

Comments?

Another problem I have is to decide the max size of the slot mem for the 
worker: proxy_lb_workers() only gives the right value after parsing the 
configuration file (in the post-config()) but the logic creates the 
workers while parsing the configuration file.

What to do? Create the slot mem with a fixed value (like 128: that is 
what I have done in my prototype)? Create the slot mem in plain memory 
and do the create of the "real" slot mem and create the workers in the 
post-config()?

Cheers

Jean-Frederoc

Re: httpd-proxy-scoreboard

[Ruediger Pluem <rp...@apache.org>]

On 07/27/2006 02:37 PM, Jean-frederic Clere wrote:
> Ruediger Pluem wrote:

>>
> You want to check all the connections of the pool corresponding to the
> worker but not all the workers.

Not all at the same time, but only the one I actually leased.

> 
>> If it is not that does not mean
>> necessarily that the server in the backend failed.
>>  
>>
> That probably means the socket has been closed ;-)

Yes, provided the backend is not faulty this will the most common case then.

> 
>> It could be just this connection. So the default response on a faulty
>> connection
>> would be to close this one and try with a new one.
>>
> With TC that means you create a new thread in TC, we must not overload
> the back-end server.

That should not be a problem in most cases. If e.g. TC has closed the socket
from its side then the thread holding the connection is free again. And if httpd
closes the connection successfully the thread also should be become available again.


Regards

Rüdiger

Re: httpd-proxy-scoreboard

[Jean-frederic Clere <jf...@gmail.com>]

Ruediger Pluem wrote:

>On 07/27/2006 11:31 AM, Jean-frederic Clere wrote:
>  
>
>>Ruediger Pluem wrote:
>>
>>    
>>
>>>On 07/26/2006 10:53 PM, Jean-frederic Clere wrote:
>>>
>>>
>>> 
>>>
>>>      
>>>
>>>>I already have a prototype of an external health checker process that
>>>>uses an AJP cping/cpong and a simple connect for http/https.
>>>>Basicaly mod_proxy uses the health_worker_method routine to write and
>>>>read from a slot mem that contains the information of the workers + a
>>>>health information. A balancer could check the health information before
>>>>using the worker. The health checker process uses the worker description
>>>>(thru the health_worker_method) to check if the back-end service is
>>>>running.
>>>>  
>>>>        
>>>>
>>>Sorry, you may have explained that before, but why using an external
>>>health
>>>checker process?
>>> 
>>>
>>>      
>>>
>>The idea is to have the health check independant from the requests so
>>that httpd knows in advance that a worker is not working.
>>    
>>
>
>Ah, now I get it. Ok, I guess we both have slightly different ideas about what
>to health check. Your idea is more of a server health check and is somewhat
>similar to what HW loadbalancers are doing: Check each backend on a regular
>schedule if it is still available.
>  
>
Yep.

>My idea is more of a connection health check. I want to check if the connection
>I leased from the pool is still healthy.
>
You want to check all the connections of the pool corresponding to the 
worker but not all the workers.

> If it is not that does not mean
>necessarily that the server in the backend failed.
>  
>
That probably means the socket has been closed ;-)

>It could be just this connection. So the default response on a faulty connection
>would be to close this one and try with a new one.
>
With TC that means you create a new thread in TC, we must not overload 
the back-end server.

> Only if this fails also then
>there seems to be something wrong with the backend server. This is more like what
>most DB connection pools offer with a health-check query.
>
>I think both ideas make sense. Of course the connection health checks should be
>simpler than the server health check to avoid too much overhead and too much
>lost time.
>That is the reason why I think that e.g. doing a HTTP request on a HTTP connection
>leased from the pool is unreasonable.
>
Yes... That is hard to check if http/https back-end server is healthy or 
not.

> We need some quick network layer check in this
>case, whereas in the AJP case CPING/CPONG has still an acceptable overhead.
>
>Finally I think if we have a good provider interface we can use the health check providers
>for both kind of health checks.
>
Right an internal httpd provide would have the 2 health checks ways and 
external one probably one.
If the health check is external we only need to process the result, if 
it is internal both health check and result processing are needed.

> We only need to create the frameworks for them
>to start in different parts of the code (separate process in your case, in my case
>I would guess that it needs to replace is_socket_connected which is called from
>ap_proxy_connect_backend)
>
>Regards
>
>Rüdiger
>
>
>  
>

Re: httpd-proxy-scoreboard

[Ruediger Pluem <rp...@apache.org>]

On 07/27/2006 11:31 AM, Jean-frederic Clere wrote:
> Ruediger Pluem wrote:
> 
>> On 07/26/2006 10:53 PM, Jean-frederic Clere wrote:
>>
>>
>>  
>>
>>> I already have a prototype of an external health checker process that
>>> uses an AJP cping/cpong and a simple connect for http/https.
>>> Basicaly mod_proxy uses the health_worker_method routine to write and
>>> read from a slot mem that contains the information of the workers + a
>>> health information. A balancer could check the health information before
>>> using the worker. The health checker process uses the worker description
>>> (thru the health_worker_method) to check if the back-end service is
>>> running.
>>>   
>>
>>
>> Sorry, you may have explained that before, but why using an external
>> health
>> checker process?
>>  
>>
> The idea is to have the health check independant from the requests so
> that httpd knows in advance that a worker is not working.

Ah, now I get it. Ok, I guess we both have slightly different ideas about what
to health check. Your idea is more of a server health check and is somewhat
similar to what HW loadbalancers are doing: Check each backend on a regular
schedule if it is still available.

My idea is more of a connection health check. I want to check if the connection
I leased from the pool is still healthy. If it is not that does not mean
necessarily that the server in the backend failed.
It could be just this connection. So the default response on a faulty connection
would be to close this one and try with a new one. Only if this fails also then
there seems to be something wrong with the backend server. This is more like what
most DB connection pools offer with a health-check query.

I think both ideas make sense. Of course the connection health checks should be
simpler than the server health check to avoid too much overhead and too much
lost time.
That is the reason why I think that e.g. doing a HTTP request on a HTTP connection
leased from the pool is unreasonable. We need some quick network layer check in this
case, whereas in the AJP case CPING/CPONG has still an acceptable overhead.

Finally I think if we have a good provider interface we can use the health check providers
for both kind of health checks. We only need to create the frameworks for them
to start in different parts of the code (separate process in your case, in my case
I would guess that it needs to replace is_socket_connected which is called from
ap_proxy_connect_backend)

Regards

Rüdiger

Re: httpd-proxy-scoreboard

[Jean-frederic Clere <jf...@gmail.com>]

Ruediger Pluem wrote:

>On 07/26/2006 10:53 PM, Jean-frederic Clere wrote:
>
>
>  
>
>>I already have a prototype of an external health checker process that
>>uses an AJP cping/cpong and a simple connect for http/https.
>>Basicaly mod_proxy uses the health_worker_method routine to write and
>>read from a slot mem that contains the information of the workers + a
>>health information. A balancer could check the health information before
>>using the worker. The health checker process uses the worker description
>>(thru the health_worker_method) to check if the back-end service is
>>running.
>>    
>>
>
>Sorry, you may have explained that before, but why using an external health
>checker process?
>  
>
The idea is to have the health check independant from the requests so 
that httpd knows in advance that a worker is not working.

>My thoughts are more the line to do a health check on a connection every time
>you lease it from the pool.
>  
>
That means you try a worker that may be down since a while (<x seconds).
If I use something request driven, tell check every x seconds I would 
like to check all workers, how to do that without slowing down the 
request that causes the health check?
I will make a try ;-)

Cheers

Jean-Frederic

>Regards
>
>Rüdiger
>
>
>  
>

Re: httpd-proxy-scoreboard

[Ruediger Pluem <rp...@apache.org>]

On 07/26/2006 10:53 PM, Jean-frederic Clere wrote:


> I already have a prototype of an external health checker process that
> uses an AJP cping/cpong and a simple connect for http/https.
> Basicaly mod_proxy uses the health_worker_method routine to write and
> read from a slot mem that contains the information of the workers + a
> health information. A balancer could check the health information before
> using the worker. The health checker process uses the worker description
> (thru the health_worker_method) to check if the back-end service is
> running.

Sorry, you may have explained that before, but why using an external health
checker process?
My thoughts are more the line to do a health check on a connection every time
you lease it from the pool.

Regards

Rüdiger

Re: httpd-proxy-scoreboard

[Jean-frederic Clere <jf...@gmail.com>]

Ruediger Pluem wrote:

>On 26.07.2006 18:11, Jean-frederic Clere wrote:
>  
>
>>Hi,
>>
>>I have started to write a "generic" health-checker for mod_proxy. I
>>would like to change the macro PROXY_WORKER_IS_USABLE() to a routine in
>>proxy_util.c.
>>
>>Comments?
>>    
>>
>
>>>From my current point of view we should keep this macro as a simple bit checker.
>+1 to your efforts writing a "generic" health-checker to mod_proxy. Do you use
>a provider approach to implement it?
>
Yes. I have attached a draft of the include I plan to use.

> That would make it possible to reimplement
>Mladen's cping_cpong for AJP as a provider.
>
I already have a prototype of an external health checker process that 
uses an AJP cping/cpong and a simple connect for http/https.
Basicaly mod_proxy uses the health_worker_method routine to write and 
read from a slot mem that contains the information of the workers + a 
health information. A balancer could check the health information before 
using the worker. The health checker process uses the worker description 
(thru the health_worker_method) to check if the back-end service is running.

> Then we would have a very flexible
>way to plugin different health-checks.
>  
>
Yep

Cheers

Jean-Frederic

>Regards
>
>Rüdiger
>
>
>  
>

Re: httpd-proxy-scoreboard

[Ruediger Pluem <rp...@apache.org>]

On 26.07.2006 18:11, Jean-frederic Clere wrote:
> Hi,
> 
> I have started to write a "generic" health-checker for mod_proxy. I
> would like to change the macro PROXY_WORKER_IS_USABLE() to a routine in
> proxy_util.c.
> 
> Comments?

>From my current point of view we should keep this macro as a simple bit checker.
+1 to your efforts writing a "generic" health-checker to mod_proxy. Do you use
a provider approach to implement it? That would make it possible to reimplement
Mladen's cping_cpong for AJP as a provider. Then we would have a very flexible
way to plugin different health-checks.

Regards

Rüdiger

Re: httpd-proxy-scoreboard

[Jim Jagielski <ji...@jaguNET.com>]

On Jul 26, 2006, at 12:11 PM, Jean-frederic Clere wrote:

> Hi,
>
> I have started to write a "generic" health-checker for mod_proxy. I  
> would like to change the macro PROXY_WORKER_IS_USABLE() to a  
> routine in proxy_util.c.
>

Why? We're simply checking bits... I can't see bothering
with the overhead of a function call.