You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modules-dev@httpd.apache.org by Alex Bligh <al...@alex.org.uk> on 2013/06/11 21:20:11 UTC
thread safety and mpm-prefork
I've written a module which I believe to be thread-safe but appears to be
doing something which I have put down to a lack of thread safety in pool
management (somewhere).
Before I tear my hair out here, my module is running with apache 2.2.22
and mpm-prefork on Ubuntu. Do the thread primatives actually do anything in
mpm-prefork? I'm using apr_thread_create to create a thread, then
providing a separate allocator, mutex, pool and similar (all as
recommended). But if the mutex stuff is 'optimised out' of my apr
library - specifically the pool stuff - all this will be in vain.
--
Alex Bligh
Re: thread safety and mpm-prefork
Posted by Alex Bligh <al...@alex.org.uk>.
On 11 Jun 2013, at 23:06, Sorin Manolache wrote:
> I'm sorry, I ran out of ideas. I suppose that the operations of the two threads on the bucket brigade are protected by mutexes...
Yep. I create my own output bucket brigade too. I *presume* this uses the allocator mutex to manipulate the bucket brigade (or that it contains its own mutex) protecting its link->next pointers.
/* We cannot use the same bucket allocator for the ouput bucket brigade
* obb as the one associated with the connection (r->connection->bucket_alloc)
* because the same bucket allocator cannot be used in two different
* threads, and we use the connection bucket allocator in this
* thread - see docs on apr_bucket_alloc_create(). This results in
* occasional core dumps. So create our own bucket allocator and pool
* for output thread bucket brigade.
*/
...
if (
( apr_thread_mutex_create(&oallocatormutex, APR_THREAD_MUTEX_UNNESTED, r->pool) == APR_SUCCESS) &&
( apr_allocator_create(&oallocator) == APR_SUCCESS) &&
( apr_allocator_mutex_set(oallocator, oallocatormutex), 1 ) &&
( apr_pool_create_ex(&opool, NULL, NULL, oallocator) == APR_SUCCESS) && /* WARNING: pool has no parent */
( NULL != (obucketallocator = apr_bucket_alloc_create(opool))) &&
( NULL != (obb = apr_brigade_create(opool, obucketallocator)))
) {
--
Alex Bligh
Re: thread safety and mpm-prefork
Posted by Sorin Manolache <so...@gmail.com>.
On 2013-06-11 23:08, Alex Bligh wrote:
> Sorin,
>
> On 11 Jun 2013, at 21:57, Sorin Manolache wrote:
>
>> The threadallocatormutex is created from a child of the request pool. The request pool and its child-pools are destroyed when the request terminates. Do you use the threadpool/threadallocator/threadallocatormutex afterwards?
>
> Nope. It's one long running request, and at the end of the request handler, the thread I've created is _join'ed, and the pools are destroyed.
>
> When I torture test it, I can run 10 hours of fullscreen video through it, and every 5 or 6th such test results in a core dump (inevitably a bucket brigade pointer being unhappy). We have a customer who seems to be talented at making things go wrong and who does not get an abort/segv, but a 100% CPU live lock. gdb suggests the destruction of the bucket brigade goes around and around - again a symptom of bucket brigade linked list pointers being unhappy.
>
I'm sorry, I ran out of ideas. I suppose that the operations of the two
threads on the bucket brigade are protected by mutexes...
S
Re: thread safety and mpm-prefork
Posted by Alex Bligh <al...@alex.org.uk>.
Sorin,
On 11 Jun 2013, at 21:57, Sorin Manolache wrote:
> The threadallocatormutex is created from a child of the request pool. The request pool and its child-pools are destroyed when the request terminates. Do you use the threadpool/threadallocator/threadallocatormutex afterwards?
Nope. It's one long running request, and at the end of the request handler, the thread I've created is _join'ed, and the pools are destroyed.
When I torture test it, I can run 10 hours of fullscreen video through it, and every 5 or 6th such test results in a core dump (inevitably a bucket brigade pointer being unhappy). We have a customer who seems to be talented at making things go wrong and who does not get an abort/segv, but a 100% CPU live lock. gdb suggests the destruction of the bucket brigade goes around and around - again a symptom of bucket brigade linked list pointers being unhappy.
--
Alex Bligh
Re: thread safety and mpm-prefork
Posted by Sorin Manolache <so...@gmail.com>.
On 2013-06-11 22:21, Alex Bligh wrote:
> Sorin,
>
> On 11 Jun 2013, at 21:10, Sorin Manolache wrote:
>
>> apr_* and mpm_prefork are different software packages and ubuntu distributes them separately. So it is almost certain that you have a thread-enabled libapr (i.e. compiled with APR_HAS_THREADS). You would not be able to compile the code that uses apr_thread_create if your libapr was not compiled with thread support.
>>
>> mpm_prefork is like any ordinary client of libapr. Just that it does not use the threading functionality in libapr. So it cannot disable/optimise out the mutexes in libapr.
>
> Thanks.
>
>> Please be aware that apr_pools are not thread-safe. Only the creation of subpools is thread-safe. So you should create a subpool per thread to stay safe.
>
> I'm doing that. In fact I'm doing:
>
> if (!( ( apr_pool_create(&pool, r->pool) == APR_SUCCESS) &&
> ( apr_thread_mutex_create(&threadallocatormutex, APR_THREAD_MUTEX_UNNESTED, pool) == APR_SUCCESS) &&
> ( apr_allocator_create(&threadallocator) == APR_SUCCESS) &&
> ( apr_allocator_mutex_set(threadallocator, threadallocatormutex), 1 ) &&
> ( apr_pool_create_ex(&threadpool, NULL, NULL, threadallocator) == APR_SUCCESS) && /* WARNING: pool has no parent\
> */
> threadpool && threadallocator && threadallocatormutex && pool
> )) {
> ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
> "tcp_proxy_on_connect could not allocate pool");
> return NULL;
> }
>
The threadallocatormutex is created from a child of the request pool.
The request pool and its child-pools are destroyed when the request
terminates. Do you use the
threadpool/threadallocator/threadallocatormutex afterwards?
> i.e. a subpool, then a new mutex, allocator, and parentless pool using the forgoing. I suspect this is well into the land of overkill.
>
> I'm doing similar with creation of bucket brigades. The issue seems to be linked to bucket brigade processing (which is unsurprising as that's what's written to by one thread but read by the other).
>
> Any help (paid if necessary) welcome. It's an apache licensed tcp proxy module.
>
Re: thread safety and mpm-prefork
Posted by Alex Bligh <al...@alex.org.uk>.
Sorin,
On 11 Jun 2013, at 21:10, Sorin Manolache wrote:
> apr_* and mpm_prefork are different software packages and ubuntu distributes them separately. So it is almost certain that you have a thread-enabled libapr (i.e. compiled with APR_HAS_THREADS). You would not be able to compile the code that uses apr_thread_create if your libapr was not compiled with thread support.
>
> mpm_prefork is like any ordinary client of libapr. Just that it does not use the threading functionality in libapr. So it cannot disable/optimise out the mutexes in libapr.
Thanks.
> Please be aware that apr_pools are not thread-safe. Only the creation of subpools is thread-safe. So you should create a subpool per thread to stay safe.
I'm doing that. In fact I'm doing:
if (!( ( apr_pool_create(&pool, r->pool) == APR_SUCCESS) &&
( apr_thread_mutex_create(&threadallocatormutex, APR_THREAD_MUTEX_UNNESTED, pool) == APR_SUCCESS) &&
( apr_allocator_create(&threadallocator) == APR_SUCCESS) &&
( apr_allocator_mutex_set(threadallocator, threadallocatormutex), 1 ) &&
( apr_pool_create_ex(&threadpool, NULL, NULL, threadallocator) == APR_SUCCESS) && /* WARNING: pool has no parent\
*/
threadpool && threadallocator && threadallocatormutex && pool
)) {
ap_log_rerror(APLOG_MARK, APLOG_DEBUG, 0, r,
"tcp_proxy_on_connect could not allocate pool");
return NULL;
}
i.e. a subpool, then a new mutex, allocator, and parentless pool using the forgoing. I suspect this is well into the land of overkill.
I'm doing similar with creation of bucket brigades. The issue seems to be linked to bucket brigade processing (which is unsurprising as that's what's written to by one thread but read by the other).
Any help (paid if necessary) welcome. It's an apache licensed tcp proxy module.
--
Alex Bligh
Re: thread safety and mpm-prefork
Posted by Sorin Manolache <so...@gmail.com>.
On 2013-06-11 21:20, Alex Bligh wrote:
> I've written a module which I believe to be thread-safe but appears to be
> doing something which I have put down to a lack of thread safety in pool
> management (somewhere).
>
> Before I tear my hair out here, my module is running with apache 2.2.22
> and mpm-prefork on Ubuntu. Do the thread primatives actually do anything in
> mpm-prefork? I'm using apr_thread_create to create a thread, then
> providing a separate allocator, mutex, pool and similar (all as
> recommended). But if the mutex stuff is 'optimised out' of my apr
> library - specifically the pool stuff - all this will be in vain.
>
apr_* and mpm_prefork are different software packages and ubuntu
distributes them separately. So it is almost certain that you have a
thread-enabled libapr (i.e. compiled with APR_HAS_THREADS). You would
not be able to compile the code that uses apr_thread_create if your
libapr was not compiled with thread support.
mpm_prefork is like any ordinary client of libapr. Just that it does not
use the threading functionality in libapr. So it cannot disable/optimise
out the mutexes in libapr.
Please be aware that apr_pools are not thread-safe. Only the creation of
subpools is thread-safe. So you should create a subpool per thread to
stay safe.
Sorin