Info about APR threads

[Richard <ju...@gmail.com>]

Hello,

I am trying to get some more info and documentation on APR threads and APR
pools. I find the documentation pretty scarce so far.
For APR pools, the link "Using APR
Pools<http://svn.apache.org/viewcvs.cgi/apr/apr/docs/pool-design.html>"
(http://svn.apache.org/viewcvs.cgi/apr/apr/docs/pool-design.html) on the APR
apache web site is unfortunately not working.
Anyhow, I've been grepping the Apache source code to find out how to use the
APR pool and APR threads API.

In the prototype I am working on, I have multi-threaded processes sharing a
shared-memory buffer (setup through a system call similar to MMAP shared). I
have to synchronize accesses to shared data found in the shared buffer. I
can actually do this using pthreads and I was the first one to be surprised.
Your pthread mutex can reside on the shared memory buffer, one process can
call ptrhead_mutex_init() and the other process can start using and locking
on that mutex.

Now APR pools greatly complicate my situation. If I undestand correctly,
apr_thread_mutex_create() needs a pool and the actual mutex will be
allocated from that pool. The location of that pool in memory will not be
shared among my processes. So is that a limitation of APR mutexes ? APR
mutexes cannot be shared among processes ? Is there a way to get around that
? Is there a hack that could be done by using a special memory allocator for
that pool ?

I am sure that the answer is that I am using the APR library for something
that it was not designed for. Unfortunately, it is hard to get a feel of how
to use a library correctly when documentation about the library is scarce.

So maybe I should give some background on what I am working on. I am working
with Apache, mod_ssl and openssl. Apache is running with the worker MPM.
I have modified Apache so that the apache processes share a shared-memory
buffer (basically setup through a syscall similar to mmap shared). I have a
special SSL engine that takes care of cerficate signatures. Each apache
thread servicing an SSL connection queues the signature request in the
shared buffer which is processed by an outside application responsible for
encryption. So I need a way to synchronize the various threads inside each
apache process.

Thanks
Richard

Re: Info about APR threads

[Richard <ju...@gmail.com>]

Thanks a lot your reply was very helpful.
I now see the 3 different types of mutexes: apr_thread mutex, apr_proc mutex
and apr_global mutex.
In my case I want to synchronize both accross threads and accross processes.
That is threadA in processX need to be syncrhonized with threadB in
processY. So I have to use APR global mutex.

I am looking at the docs found on this page:
http://apr.apache.org/docs/apr/group___a_p_r___global_mutex.html#ga1
if you could clarify some questions I have that would be great:

1.) apr_global_mutex_create() should be called only once by the first
process to use the mutex. Then, subsequent processes that want to use the
mutex should call the function apr_global_mutex_child_init().
2.) The pool argument passed to apr_global_mutex_create() and
apr_global_mutex_child_init() in question#1 above is just a regular pool
that is local to each process.
3.) When everything is done, each process has to invoke
apr_global_mutex_destroy().

I have a feeling that this global mutex will be very expensive and is using
some locking involving files.

You are right, my SSL engine is a critical point. The SSL signature will not
be done by a hardware crypto accelerator but by another piece of software.
My SSL engine will batch SSL signatures in a shared queue before sending
them off to the application responsible for crypto services, so that is why
I need some locking on that shared queue of outstanding SSL requests. I am
wondering how SSL engines for hardware crypto accelerators deal with
asynchronous accesses ? I had the feeling that the actual read and write
syscall to the hardware's device driver will take care of serializing the
SSL requests. As a matter of fact, I am now wondering how the default
builtin SSLeay engine deals with concurrency ? My guess is that the default
builtin SSLeay engine does not have to worry about cross-process
concurrency. It only has to deal with cross-thread concurrency and hence can
use the more light-weight apr_thread_mutexes.

Thanks
Richard


On 12/21/05, Nick Kew <ni...@webthing.com> wrote:
>
> On Wednesday 21 December 2005 11:09, Richard wrote:
>
> > Now APR pools greatly complicate my situation. If I undestand correctly,
> > apr_thread_mutex_create() needs a pool and the actual mutex will be
> > allocated from that pool.
>
> Yep.
>
> > The location of that pool in memory will not be
> > shared among my processes.
>
> Erm, threads or processes?  If the former, that's moot.  If the latter,
> you would want an APR global mutex.
>
> > So is that a limitation of APR mutexes ? APR
> > mutexes cannot be shared among processes ? Is there a way to get around
> > that ? Is there a hack that could be done by using a special memory
> > allocator for that pool ?
>
> That's why there are three different types of mutex in APR.





> So maybe I should give some background on what I am working on. I am
> > working with Apache, mod_ssl and openssl. Apache is running with the
> worker
> > MPM. I have modified Apache so that the apache processes share a
> > shared-memory buffer (basically setup through a syscall similar to mmap
> > shared).
>
> The conventional Apache way to do that would be with apr_shm.
>
> > I have a special SSL engine that takes care of cerficate
> > signatures. Each apache thread servicing an SSL connection queues the
> > signature request in the shared buffer which is processed by an outside
> > application responsible for encryption. So I need a way to synchronize
> the
> > various threads inside each apache process.
>
> Sounds like the critical point is how you communicate with the external
> application.  Can that be fixed to support asynchronous access?
>
> --
> Nick Kew
>

Re: Info about APR threads

[Nick Kew <ni...@webthing.com>]

On Wednesday 21 December 2005 11:09, Richard wrote:

> Now APR pools greatly complicate my situation. If I undestand correctly,
> apr_thread_mutex_create() needs a pool and the actual mutex will be
> allocated from that pool.

Yep.

> The location of that pool in memory will not be 
> shared among my processes.

Erm, threads or processes?  If the former, that's moot.  If the latter,
you would want an APR global mutex.

> So is that a limitation of APR mutexes ? APR 
> mutexes cannot be shared among processes ? Is there a way to get around
> that ? Is there a hack that could be done by using a special memory
> allocator for that pool ?

That's why there are three different types of mutex in APR.

> So maybe I should give some background on what I am working on. I am
> working with Apache, mod_ssl and openssl. Apache is running with the worker
> MPM. I have modified Apache so that the apache processes share a
> shared-memory buffer (basically setup through a syscall similar to mmap
> shared).

The conventional Apache way to do that would be with apr_shm.

> I have a special SSL engine that takes care of cerficate 
> signatures. Each apache thread servicing an SSL connection queues the
> signature request in the shared buffer which is processed by an outside
> application responsible for encryption. So I need a way to synchronize the
> various threads inside each apache process.

Sounds like the critical point is how you communicate with the external 
application.  Can that be fixed to support asynchronous access?

-- 
Nick Kew

Re: Info about APR threads

[Garrett Rooney <ro...@electricjellyfish.net>]

On 12/21/05, Richard <ju...@gmail.com> wrote:

>  For APR pools, the link "Using APR Pools"
> (http://svn.apache.org/viewcvs.cgi/apr/apr/docs/pool-design.html
> ) on the APR apache web site is unfortunately not working.

Looks like that link dates from the CVS days.  The current working
link would be:

http://svn.apache.org/viewcvs.cgi/*checkout*/apr/apr/trunk/docs/pool-design.html

I'll update the site.

-garrett