Mpm maintenance thread hook

[Mladen Turk <mt...@apache.org>]

Hi,

Right now any event happening in mpm child process
is per request based. As an example I'll give
mod_proxy and its connection pool. The connection_pool
maintenance happens only when a request comes in which
is fine for a frequently accessed server. If there
is no request for a long time the ttl (time to live)
is useless cause nothing can call the maintenance.

I propose we create a single maintenance thread (configurable)
for each child process (on creation) with child lifetime,
and configurable interval that will call module callbacks
registered to this hook.

Something like in register hooks:
ap_hook_watchdog(watchdog_handler, NULL, NULL, APR_HOOK_FIRST);

The maintenance (watchdog) thread would call registered
hooks at configured interval independent from the request
rate allowing modules to do what ever needed.

Usage examples:
mod_proxy connection pool maintenance
mod_balancer decaying (averaging) the transferred statistics,
              so that node-up/recovery doesn't consume all the load.
mod_proxy_ajp sending ping/pong at regular intervals (detecting
               node failure before request)
mod_cache probably lots of use cases since its time interval related

I'm not sure if the hook is a correct API to choose or
something like ap_register_provider API that will allow
module to register the desired watchdog interval, or something.

Right now this can be done on per-module basis by creating
the thread on child_init (we have that in mod_jk for example),
but this can lead to multiple threads created by modules
basically only calling some function at regular intervals.
Having that server wide would lower down the resource usage
although it might lead to serialization problems if the
callback takes a long time to execute.

Comments, other ideas?

Regards
-- 
^(TM)

Re: Mpm maintenance thread hook

[Jim Jagielski <ji...@jaguNET.com>]

On Aug 4, 2008, at 1:57 PM, Paul Querna wrote:

> Mladen Turk wrote:
>> Comments, other ideas?
>
> I would prefer to not add a separate 'dedicated' thread, but just  
> 'use' a worker thread, when it is about to idle instead.  When you  
> are about to go into the idle wait, check if there are any timed  
> tasks that need to execute soon, and run that instead of going idle.
>

+1

Re: Mpm maintenance thread hook

[Graham Leggett <mi...@sharp.fm>]

Mladen Turk wrote:

> IMO this should be just like any listener thread, so that logic
> can be decoupled from the main worker logic serving requests.

+1.

I looked at the monitor hook, which I understand to run every 10 seconds 
or so, though that may not be convenient for everybody. The monitor hook 
is passed the pool used to configure the server, so you could tag that 
pool with data to process, but it is restrictive.

Regards,
Graham
--

Re: Mpm maintenance thread hook

[Mladen Turk <mt...@apache.org>]

Paul Querna wrote:
> Mladen Turk wrote:
>> Comments, other ideas?
> 
> I would prefer to not add a separate 'dedicated' thread, but just 'use' 
> a worker thread, when it is about to idle instead.  When you are about 
> to go into the idle wait, check if there are any timed tasks that need 
> to execute soon, and run that instead of going idle.
> 

Since this needs to happen on regular intervals then there has to
be some 'dedicated' thread that will at least pop the idle worker
thread from the pool. I don't see any benefit for adding additional
logic to worker threads that will figure out the reason for wakeup
(accepted socket or task event)
Well instead this thread we could modify the child main to wait both
on exit event and task interval, but this again complicates the
things much more then it should.

IMO this should be just like any listener thread, so that logic
can be decoupled from the main worker logic serving requests.


Regards
-- 
^(TM)

Re: Mpm maintenance thread hook

[Paul Querna <ch...@force-elite.com>]

Mladen Turk wrote:
> Comments, other ideas?

I would prefer to not add a separate 'dedicated' thread, but just 'use' 
a worker thread, when it is about to idle instead.  When you are about 
to go into the idle wait, check if there are any timed tasks that need 
to execute soon, and run that instead of going idle.

This is where I wanted to do with the (unfinished) Simple MPM...  Have 
an API to schedule callbacks in the future, react to IO, or other events.

Rather than having dedicated threads for all of them, I wanted to use a 
single worker pool.

-Paul

Re: Mpm maintenance thread hook

[Jim Jagielski <ji...@jaguNET.com>]

On Aug 6, 2008, at 5:16 PM, Akins, Brian wrote:

> On 8/4/08 3:41 PM, "Jim Jagielski" <ji...@jaguNET.com> wrote:
>
>> *grin*
>>
>> I've always wondered what usefulness
>>
>>        ap_run_monitor(p)
>
> FWIW, we use the monitor hook all the time...

The usefulness of being passed "just a pool" is what I
was referring to ;)

Re: Mpm maintenance thread hook

["Akins, Brian" <Br...@turner.com>]

On 8/4/08 3:41 PM, "Jim Jagielski" <ji...@jaguNET.com> wrote:

> *grin*
> 
> I've always wondered what usefulness
> 
>         ap_run_monitor(p)

FWIW, we use the monitor hook all the time... Most of our stuff uses some
shared memory and the monitor hook in parent, but I have considered hacking
something into each child (I may have actually done that before...).  We
rarely have overly idle children, so we do have something that does like
what Paul was suggesting.  We used one of the really late connection hooks
(I think, I'm away from my normal computing spot), so we didn't hack the
MPM, but what Paul suggests sound plausible.  Of course, we run more threads
per child than a lot of people, so an extra "utility" thread would not be
such a big deal and may keep some stuff "cleaner."

-- 
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies

Re: Mpm maintenance thread hook

[Jim Jagielski <ji...@jaguNET.com>]

On Aug 4, 2008, at 1:25 PM, Akins, Brian wrote:

> On 8/4/08 12:59 PM, "Mladen Turk" <mt...@apache.org> wrote:
>
>> mod_balancer decaying (averaging) the transferred statistics,
>>              so that node-up/recovery doesn't consume all the load.
>> mod_proxy_ajp sending ping/pong at regular intervals (detecting
>>               node failure before request)
>> mod_cache probably lots of use cases since its time interval related
>
>
> Couldn't these be in the monitor hook in parent, assuming some  
> shared memory
> tied the children together?
>

*grin*

I've always wondered what usefulness

	ap_run_monitor(p)

was... 

Re: Mpm maintenance thread hook

["Akins, Brian" <Br...@turner.com>]

On 8/4/08 12:59 PM, "Mladen Turk" <mt...@apache.org> wrote:

> mod_balancer decaying (averaging) the transferred statistics,
>               so that node-up/recovery doesn't consume all the load.
> mod_proxy_ajp sending ping/pong at regular intervals (detecting
>                node failure before request)
> mod_cache probably lots of use cases since its time interval related


Couldn't these be in the monitor hook in parent, assuming some shared memory
tied the children together?

I'm +1 on the idea.  For 99% of my stuff, the servers get access frequently
enough that it doesn't matter, but I have had to hack together a few child
maintenance threads.


-- 
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies