per-worker-thread counter question

[Neil McKee <ne...@inmon.com>]

Hi,

Here's an easy question for someone who knows their way around...

I want to maintain a new global counter,  but for performance reasons I am reluctant to use a mutex or atomic_increment to update it.  I would rather maintain a separate counter for every worker-thread,  and only accumulate the global counter when required.  (If the per-worker-thread counter is 32-bit then I shouldn't even need a mutex when accumulating the total across all the current threads).

Obviously I shouldn't just declare something as "__thread apr_int32_t mycounter;" and mince it together as a linux-only hack.  I'd like to find the portable apr-library way to do it.   So I think I need to find the following:

* - a hook that is called whenever a worker thread is started
* - a hook that is called whenever a worker thread is about to die
* - a hook to find_or_create a 32-bit integer that is private to the current worker-thread
* - a fn to iterate (safely) over all the current worker threads

It's the last one that seems particularly elusive.  I could't find an ap_ or apr_ library call that seemed to do anything like that.

If this has all been done before,   please can you point me to the relevant module sources?  I think it would save me a lot of time.   Alternatively,  if you think I should just relax and use an atomic increment instead,  then let me know.

Thanks!

Neil

Re: per-worker-thread counter question

[Nick Kew <ni...@apache.org>]

On 28 Jun 2011, at 03:57, Neil McKee wrote:

> Hi,
> 
> Here's an easy question for someone who knows their way around...

Yeah, right :P 

> I want to maintain a new global counter,  but for performance reasons I am reluctant to use a mutex or atomic_increment to update it.  I would rather maintain a separate counter for every worker-thread,  and only accumulate the global counter when required.  (If the per-worker-thread counter is 32-bit then I shouldn't even need a mutex when accumulating the total across all the current threads).

Have you tried apr_atomic and found an unacceptable performance impact,
or are you just assuming it?  I haven't tried it, but my gut feeling wouldn't
expect the overhead to be out of proportion.

> Obviously I shouldn't just declare something as "__thread apr_int32_t mycounter;" and mince it together as a linux-only hack.  I'd like to find the portable apr-library way to do it.   So I think I need to find the following:
> 
> * - a hook that is called whenever a worker thread is started
> * - a hook that is called whenever a worker thread is about to die
> * - a hook to find_or_create a 32-bit integer that is private to the current worker-thread
> * - a fn to iterate (safely) over all the current worker threads

Absent something in the API, you're stuck.  Just to make it work across
standard MPMs will require care, and is unlikely to be future-proof.

> It's the last one that seems particularly elusive.  I could't find an ap_ or apr_ library call that seemed to do anything like that.

So your module would need to maintain its own register to iterate over.
Hats off if you can hack that with less overhead than apr_atomic or ...

> If this has all been done before,   please can you point me to the relevant module sources?  I think it would save me a lot of time.   Alternatively,  if you think I should just relax and use an atomic increment instead,  then let me know.

What springs to mind (as being supported in the API) is the scoreboard.
mod_slotmem (new in trunk) may be worth a look.

-- 
Nick Kew

Available for work, contract or permanent
http://www.webthing.com/~nick/cv.html

Re: per-worker-thread counter question

[Neil McKee <ne...@inmon.com>]

Trying to iterate over worker threads that may be starting and stopping seemed just too darn hard (for me anyway).  Reading the "scoreboard" code put me off.   I ended up using atomic operations.  In the mod_sflow module this boils down to just 3 atomic operations in the critical path.  That's two calls to apr_atomic_inc32() to bump global counters and one apr_atomic_dec32() to see if the transaction should be sampled (see mod-sflow.googlecode.com).   This way the module doesn't know or care how many threads there are.

Although "apr_atomic_inc32(&counter)" is much more expensive than "counter++"  my understanding is that this is still much better than having threads stall as they wait for a contested mutex.   Someone please tell me if I got that wrong(!)

For a truly global counter you still have to aggregate across multiple unix processes.  A shared memory segment might be the  best way to do that  but I wasn't sure if that was really going to work with atomic operations.   In any event I found it convenient to use a pipe.  If you open a pipe before the child-forking starts then every child process automatically sees the same pipe and they can all send messges up to the "master" who is still reading from the other end of it.   As long as those messages are less than PIPE_BUF in length (sometimes 512,  but usually 4096 bytes,  depending on the OS) then they are atomic and don't get interleaved.  This way the "master" doesn't  know or care how many child processes there are.

Neil


On Jun 30, 2011, at 2:31 AM, Massimo Manghi wrote:

> I found this problem conceptually interesting and worth of generalization in many contexts. I'd like to know if Neil got around the problem with something that could be a satisfying solution.
> 
> thanks
> 
>  -- Massimo
> 
> On 06/28/2011 04:57 AM, Neil McKee wrote:
>> Hi,
>> 
>> Here's an easy question for someone who knows their way around...
>> 
>> I want to maintain a new global counter,  but for performance reasons I am reluctant to use a mutex or atomic_increment to update it.  I would rather maintain a separate counter for every worker-thread,  and only accumulate the global counter when required.  (If the per-worker-thread counter is 32-bit then I shouldn't even need a mutex when accumulating the total across all the current threads).
>> 
>> Obviously I shouldn't just declare something as "__thread apr_int32_t mycounter;" and mince it together as a linux-only hack.  I'd like to find the portable apr-library way to do it.   So I think I need to find the following:
>> 
>> * - a hook that is called whenever a worker thread is started
>> * - a hook that is called whenever a worker thread is about to die
>> * - a hook to find_or_create a 32-bit integer that is private to the current worker-thread
>> * - a fn to iterate (safely) over all the current worker threads
>> 
>> It's the last one that seems particularly elusive.  I could't find an ap_ or apr_ library call that seemed to do anything like that.
>> 
>> If this has all been done before,   please can you point me to the relevant module sources?  I think it would save me a lot of time.   Alternatively,  if you think I should just relax and use an atomic increment instead,  then let me know.
>> 
>> Thanks!
>> 
>> Neil
> 

Re: per-worker-thread counter question

[Massimo Manghi <ma...@unipr.it>]

I found this problem conceptually interesting and worth of 
generalization in many contexts. I'd like to know if Neil got around the 
problem with something that could be a satisfying solution.

thanks

   -- Massimo

On 06/28/2011 04:57 AM, Neil McKee wrote:
> Hi,
>
> Here's an easy question for someone who knows their way around...
>
> I want to maintain a new global counter,  but for performance reasons I am reluctant to use a mutex or atomic_increment to update it.  I would rather maintain a separate counter for every worker-thread,  and only accumulate the global counter when required.  (If the per-worker-thread counter is 32-bit then I shouldn't even need a mutex when accumulating the total across all the current threads).
>
> Obviously I shouldn't just declare something as "__thread apr_int32_t mycounter;" and mince it together as a linux-only hack.  I'd like to find the portable apr-library way to do it.   So I think I need to find the following:
>
> * - a hook that is called whenever a worker thread is started
> * - a hook that is called whenever a worker thread is about to die
> * - a hook to find_or_create a 32-bit integer that is private to the current worker-thread
> * - a fn to iterate (safely) over all the current worker threads
>
> It's the last one that seems particularly elusive.  I could't find an ap_ or apr_ library call that seemed to do anything like that.
>
> If this has all been done before,   please can you point me to the relevant module sources?  I think it would save me a lot of time.   Alternatively,  if you think I should just relax and use an atomic increment instead,  then let me know.
>
> Thanks!
>
> Neil