thread termination vs. pool lifetime

[David Pope <da...@citrix.com>]

Hello all,

I'm writing a module that launches a background thread that periodically
wakes up to do some maintenance work.  This thread is launched for the
child process (all references in this email are to the winnt version
of the code, but the problem space may be more general).  I need to
gracefully terminate this thread as the child process exits.

Here's the "obvious" solution (mentioned in December in response to a
similar problem):  Launch the thread in child_init, and register a
cleanup with pchild to terminate the thread.

AFAICS this doesn't really work, due to a combination of the way pool
cleanups work and the way APR threads use pools:

    - A pool's cleanups are run AFTER all of its child pools have been
      destroyed.
    - apr_thread_create() creates a child pool for internal use of
      the thread logic.
    - apr_thread_exit() calls apr_pool_destroy() on this pool.

So the problem is that by the time the cleanup is run, the per-thread
pool has already been destroyed; apr_thread_exit() tries to destroy
it again, and *boom*.

The first question is, did I miss something obvious that could make this
work?

Assuming no:  I can see several ways out of this problem, but I'm very
much the Apache newbie and would appreciate some advice, and perhaps
some history behind why things were done the way they were.  There is
frequently good reasoning hidden behind hard-to-follow implementation
details.

Possible solutions:

    - Use apr_pool_parent_get() to get pconf out of pchild, using that
      to launch the thread; but register the termination callback with
      pchild, which guarantees it will run while pconf is still valid.
      This solution is awkward and nonintuitive, and not really
      future-proof.  (My familiarity with non-winnt architectures is
      limited, but I suspect this may not even an option, if pconf
      lives in shared memory or otherwise has out-of-proc limitations.)

    - Modify apr_pool_destroy to run cleanups BEFORE destroying child
      pools.  This would pretty much reverse the order of cleanup
      execution in a pool hierarchy, which is probably a Bad Thing.
      I assume there is a good reason why this order (destroy children
      before running cleanups) was chosen; can someone provide some
      background?

    - I don't really need the return value from the thread, so I could
      technically get away with not calling apr_thread_exit().  I don't
      consider this to be a real solution, and probably won't work
      at all for pthreads-based systems.

    - Add a child_shutdown hook to the MPM that is run just before
      pchild is destroyed, which is used instead of the pool cleanup.

The fourth solution is the one I prefer; it has the following advantages:

    - It is symmetrical.  child_init is run immediately after pchild
      is created, so running child_shutdown immediately before its
      destruction makes sense.  It guarantees that the entire pchild
      hierarchy is valid during the hook.  Since pchild was valid
      during child_init, its makes sense for it to be valid during
      child_shutdown.

    - It removes ordering constraints for cleanups.  In my implementation
      I create mutexes in the same pool that I create my thread in
      (seems like a reasonable thing to do, especially if it's pchild).
      If I use a pool cleanup for thread termination, I have to make
      sure my thread termination cleanup is registered LAST, so it's
      executed FIRST.  Otherwise, the mutexes my thread uses will be
      gone by the time it tries to acquire them.  This pool cleanup
      approach would be bad because it relies on the internal
      implementation of the cleanup logic, plus it's nonintuitive.

    - It doesn't touch anything else in the system, and is reasonably
      future-proof compared to the first solution; if someone someday
      decides to change the way pconf fits into the system, this part
      of the system won't be affected.

Does this sound like a reasonable solution?

I apologize for the length; this is a complicated problem.

-- Dave
----
David Pope
Citrix Systems, Inc.