Creating an allocator with no mutex...

[Brad Nicholes <BN...@novell.com>]

   Under what circumstances would you want to create an allocator
without a mutex assigned to it?  We have been running into a problem
with the NetWare MPM on multi-processor boxes where Apache faults
periodically while trying to destroy the memory pool.  The fault appears
to be caused by a corrupt pool list.  It has all of the signs of two
threads manipulating the same list at the same time because the list was
not properly protected.  What we discovered was that due to the
following code that is executed when each worker_thread is started, the
per-thread memory pool is created based on an allocator that does not
hold a mutex.

    apr_allocator_create(&allocator);
    apr_allocator_max_free_set(allocator, ap_max_mem_free);

    apr_pool_create_ex(&pthrd, pmain, NULL, allocator);
    apr_allocator_owner_set(allocator, pthrd);
    apr_pool_tag(pthrd, "worker_thrd_pool");

    apr_pool_create(&ptrans, pthrd);
    apr_pool_tag(ptrans, "transaction");

    bucket_alloc = apr_bucket_alloc_create(pthrd);

After looking through many of the other MPM's, it appears that most of
them are doing something similar.  In other words, the per-thread pool
that is created when each threads is spun up, is based on an allocator
that does not hold a mutex.  This means that all pool creation,
destruction or manipulation is not thread safe.  The bigger question is,
shouldn't creating an allocator mutex happen automatically whenever it
is running on a threaded OS?

Brad 

Brad Nicholes
Senior Software Engineer
Novell, Inc., the leading provider of Net business solutions
http://www.novell.com 

Re: Creating an allocator with no mutex...

[Cliff Woolley <jw...@virginia.edu>]

On Fri, 18 Jul 2003, Brad Nicholes wrote:

>    Under what circumstances would you want to create an allocator
> without a mutex assigned to it?  We have been running into a problem
> with the NetWare MPM on multi-processor boxes where Apache faults
> periodically while trying to destroy the memory pool.  The fault appears

Jean-Jacques had written me off-list about the same problem a little while
ago, and I just wrote back to him about that.  You create an allocator
with no mutex on it for a pool that will only ever be used to allocate
entities (including subpools) in the current thread.  In the other MPM's,
it's true that each thread pool has no allocator mutex -- because that
thread pool will only ever allocate things for that one thread.  But the
thread pool's parent, which is the process pool in most other MPM's, does
have a mutex on it, meaning that sibling lists which are used when
creating and destroying new threads /are/ protected by mutex.  I'm
guessing that what you're seeing is that, since netware has no processes,
the one thread that handles creation of other threads has a thread pool
(of which the other threads' pools are children) that is missing the mutex
it ought to have.

I could be wrong though, as my cursory glance through worker and
the netware mpm just now did not reveal where the mutex ought to be set.
Unfortunately I'm on my way out of town for the weekend right now or I'd
look into it more.

Hopefully Sander can fill in to the extent that I'm full of shit.  :)

--Cliff