Question re design/usage of lock/mutex routines

[Tim Bray <tb...@textuality.com>]

I'm laying out plans for a big apache module and figuring out how to do 
locking/synchronization for in-memory data structures.  Classical 
multiple-reader/one-writer architecture, easy to implement with one 
atomic & one counting semaphore.

So I'm poking through APR docs, and I find obvious candidates:  Global 
Locking Routines (apr_global_mutex_child_init etc), Process Locking 
Routines (apr_proc_mutex_create etc), Thread Mutex Routines 
(apr_thread_mutex_create etc), and Thread RWLock Routines 
(apr_thread_rwlock_create etc).

Q0: Is there a better place to ask this kind of question or a TFM that I 
can R?  I spent some time in http://apr.apache.org/docs/apr/groupMumble

Q1: I assume global locks dispatch to Process or Thread locks as 
appropriate, with equivalent semantics?

Q2: How do the global routines decide which to call?

Q3: There is a stern warning in the global locking routines "There is 
considerable overhead in using this API if only cross-process or 
cross-thread mutual exclusion is required."  Suppose I want to build an 
apache module that will run in both threaded and forking deployments; do 
I shut up and swallow the overhead?  Or is this question evidence that 
I'm Unclear On The Concept?

Q4: I really need more or less exactly the RWLock semantics.  Is there 
any particular reason this is thread-only?  With ref to Q3 above, does 
this suggest I'd better build the semantics based on the global locks?

Cheers, Tim

Re: Question re design/usage of lock/mutex routines

[Aaron Bannert <aa...@clove.org>]

On Thu, Jun 13, 2002 at 04:54:20PM -0700, Tim Bray wrote:
> Ryan Bloom wrote:
> >Pretty much.  The biggest difference is that on many platforms, the
> >process locks are global in and of themselves.  In that case, we just
> >use a process lock.
> 
> I'm not 100% clear on the sense of "global" here.

Global means that you are guaranteed to have a mutex that can protect
you across threads and processes at the same time. A process lock does
not guarantee that threads in the same process will not enter the
critical at the same time. A thread lock only works within one process.
Therefore a global mutex was created to unify these two concepts.
(On many platforms, as Ryan pointed out, a process mutex encompasses
both concepts, but on some it doesn't, therefore we make a distinction.)

> Which suggests that if you're running threaded, you *really* win by 
> using the apr_thread_mutex_* stuff?  Seems it be nice if there was a way 
> for the global routines to detect which of process/thread is appropriate 
> from the httpd config or some other environmental condition?  Or even on 
> explicit init()-time signal from the caller?  Seems klunky to have all 
> these triples of routines (thread/proc/global) with identical semantics.

The proper way to think about this is in terms of the resource you are
trying to protect. If you have a resource that is only shared between
threads (ignore the MPM for now), then you only need an apr_thread_mutex.
If you plan on having a resource stored in shared memory and used by
threads in multiple processes then you'll have to go with an
apr_global_mutex.

If you plan for a shared-memory resource and use a global_mutex,
but end up having your module being run under the prefork MPM, then
you really don't have to worry about the overhead from the contention
of the extra thread-mutex hidden in this global_mutex. In all cases
that I've seen the overhead is negligible.

> >>Q3: There is a stern warning in the global locking routines "There is
> >>considerable overhead in using this API
> >
> >The warning is too stern.  If you want to work in a process/thread
> >environment...
> 
> Which I always do if I want to run in httpd in the general case, right?

I agree with Ryan here, this warning should be changed to be less stern.

> >The reason it doesn't currently exist, is that nobody has gotten around
> >to implementing it.  :-)  
> 
> Good enough - if we do the module I'll do it; I agree it doesn't feel 
> like rocket science, particularly given the number of pieces that are 
> already there.   -Tim

It's not entirely that simple, since you really need to get some sort
of cross-process scheduling mechanism. You also will have to deal with
fair queuing issues, such as a slow stream of writers keeping readers
from doing their work. IMHO the reason this isn't implemented in
a cross-process context is because there is no good support at the OS
level in all of the platforms that we wish to support in APR.

-aaron

RE: Question re design/usage of lock/mutex routines

[Ryan Bloom <rb...@covalent.net>]

> From: Tim Bray [mailto:tbray@textuality.com]
> 
> Ryan Bloom wrote:
> 
> >>Q1: I assume global locks dispatch to Process or Thread locks as
> >>appropriate, with equivalent semantics?
> >
> > Pretty much.  The biggest difference is that on many platforms, the
> > process locks are global in and of themselves.  In that case, we
just
> > use a process lock.
> 
> I'm not 100% clear on the sense of "global" here.

A global lock simply means that all processes and threads are locked.

> >>Q2: How do the global routines decide which to call?
> >
> > There is no real decision to make.  The global locks always use a
> > process lock, if you configured APR with threads and the platform
does
> > not automatically lock threads when using a process lock, then a
thread
> > lock is also used.
> 
> Which suggests that if you're running threaded, you *really* win by
> using the apr_thread_mutex_* stuff?  Seems it be nice if there was a
way
> for the global routines to detect which of process/thread is
appropriate
> from the httpd config or some other environmental condition?  Or even
on
> explicit init()-time signal from the caller?  Seems klunky to have all
> these triples of routines (thread/proc/global) with identical
semantics.

The type of lock you use depends on what you are trying to do, and what
kind of mutex protection you need.  It doesn't make any sense to use a
global lock if you just want to lock threads.  On the other hand, if you
want to lock all threads and all processes, then using a global lock is
the best possible answer.

> >>Q3: There is a stern warning in the global locking routines "There
is
> >>considerable overhead in using this API
> >
> > The warning is too stern.  If you want to work in a process/thread
> > environment...
> 
> Which I always do if I want to run in httpd in the general case,
right?

Yes.

Ryan

Re: Question re design/usage of lock/mutex routines

[Tim Bray <tb...@textuality.com>]

Ryan Bloom wrote:

>>Q0: Is there a better place to ask this kind of question
> 
> This is the right place.  

Heh-heh.  That means you get follow-ups.

>>Q1: I assume global locks dispatch to Process or Thread locks as
>>appropriate, with equivalent semantics?
> 
> Pretty much.  The biggest difference is that on many platforms, the
> process locks are global in and of themselves.  In that case, we just
> use a process lock.

I'm not 100% clear on the sense of "global" here.

>>Q2: How do the global routines decide which to call?
> 
> There is no real decision to make.  The global locks always use a
> process lock, if you configured APR with threads and the platform does
> not automatically lock threads when using a process lock, then a thread
> lock is also used.

Which suggests that if you're running threaded, you *really* win by 
using the apr_thread_mutex_* stuff?  Seems it be nice if there was a way 
for the global routines to detect which of process/thread is appropriate 
from the httpd config or some other environmental condition?  Or even on 
explicit init()-time signal from the caller?  Seems klunky to have all 
these triples of routines (thread/proc/global) with identical semantics.

>>Q3: There is a stern warning in the global locking routines "There is
>>considerable overhead in using this API
> 
> The warning is too stern.  If you want to work in a process/thread
> environment...

Which I always do if I want to run in httpd in the general case, right?

>>Q4: I really need more or less exactly the RWLock semantics.  Is there
>>any particular reason this is thread-only? 
> 
> The reason it doesn't currently exist, is that nobody has gotten around
> to implementing it.  :-)  

Good enough - if we do the module I'll do it; I agree it doesn't feel 
like rocket science, particularly given the number of pieces that are 
already there.   -Tim

RE: Question re design/usage of lock/mutex routines

[Ryan Bloom <rb...@covalent.net>]

> From: Tim Bray [mailto:tbray@textuality.com]
>
> So I'm poking through APR docs, and I find obvious candidates:  Global
> Locking Routines (apr_global_mutex_child_init etc), Process Locking
> Routines (apr_proc_mutex_create etc), Thread Mutex Routines
> (apr_thread_mutex_create etc), and Thread RWLock Routines
> (apr_thread_rwlock_create etc).
> 
> Q0: Is there a better place to ask this kind of question or a TFM that
I
> can R?  I spent some time in
http://apr.apache.org/docs/apr/groupMumble

This is the right place.  Unfortunately, the docs aren't great
throughout APR, so people generally have to ask questions on this list.
If you want to doc what you learn, that would be fantastic.

> Q1: I assume global locks dispatch to Process or Thread locks as
> appropriate, with equivalent semantics?

Pretty much.  The biggest difference is that on many platforms, the
process locks are global in and of themselves.  In that case, we just
use a process lock.

> Q2: How do the global routines decide which to call?

There is no real decision to make.  The global locks always use a
process lock, if you configured APR with threads and the platform does
not automatically lock threads when using a process lock, then a thread
lock is also used.

> Q3: There is a stern warning in the global locking routines "There is
> considerable overhead in using this API if only cross-process or
> cross-thread mutual exclusion is required."  Suppose I want to build
an
> apache module that will run in both threaded and forking deployments;
do
> I shut up and swallow the overhead?  Or is this question evidence that
> I'm Unclear On The Concept?

The warning is too stern.  If you want to work in a process/thread
environment, you want to use a global lock.  The warning is there,
because many people default to global locks when a process or thread
lock will do.  For those people, the warning is appropriate.

> Q4: I really need more or less exactly the RWLock semantics.  Is there
> any particular reason this is thread-only?  With ref to Q3 above, does
> this suggest I'd better build the semantics based on the global locks?

The reason it doesn't currently exist, is that nobody has gotten around
to implementing it.  :-)  In reality, the thread RWLocks were easy, so
they were done quickly.  If you want to implement process RWLocks, then
we would welcome the contribution.

Ryan