You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Justin Erenkrantz <je...@apache.org> on 2002/04/15 08:16:15 UTC
Segfaults restarting on Linux with worker MPM
While hunting down some PRs, I find that I keep segfaulting with
worker MPM on Linux 2.4 when I restart (sending it a SIGHUP).
(gdb) where
#0 __pthread_mutex_lock (mutex=0x410) at mutex.c:99
#1 0x40314098 in free () from /lib/libc.so.6
#2 0x40045595 in apr_allocator_destroy (allocator=0x81d1210)
at apr_pools.c:135
#3 0x40018a55 in apr_bucket_alloc_destroy (list=0x81d1288)
at apr_buckets_alloc.c:114
#4 0x080b0177 in worker_thread (thd=0x8143270, dummy=0x819c398)
at worker.c:877
#5 0x4003f770 in dummy_worker (opaque=0x8143270) at thread.c:127
#6 0x402880ae in pthread_start_thread (arg=0xbc3ffc00) at manager.c:291
Can anyone else reproduce this? This is an SMP box, so maybe there
is a race condition on shutdown? -- justin
Re: Segfaults restarting on Linux with worker MPM
Posted by Cliff Woolley <jw...@virginia.edu>.
On Mon, 15 Apr 2002, David Ford wrote:
> Just a side note, I use a wrapper for free() in my code, it checks to
> ensure the input isn't 0x0, frees the passed pointer, then sets the
> pointer to 0x0. If the passed pointer is already 0x0, I log where it
> was called from so my logs show I'm doing a double free. The added
In this case, we have multiple references to the same object, so setting
one pointer to NULL wouldn't stop us from trying to free the other one.
:)
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Segfaults restarting on Linux with worker MPM
Posted by David Ford <da...@blue-labs.org>.
Just a side note, I use a wrapper for free() in my code, it checks to
ensure the input isn't 0x0, frees the passed pointer, then sets the
pointer to 0x0. If the passed pointer is already 0x0, I log where it
was called from so my logs show I'm doing a double free. The added
benefit of free(0x0) in itself on most platforms, it simply returns.
-d
Justin Erenkrantz wrote:
>On Mon, Apr 15, 2002 at 07:17:39AM -0400, Jeff Trawick wrote:
>
>
>>backtrace in that post. A rarer backtrace is where an internal free()
>>routine segfaults (perhaps similar to your backtrace, where some heap
>>structures are bogus, probably due to a bad free()?).
>>
>>
>
>I bet it could be related to a double-free condition. That would
>make the most sense (to me).
>
>I'm not sure which commit Cliff is talking about to OS/2.
>
>Cliff? -- justin
>
>
Re: Segfaults restarting on Linux with worker MPM
Posted by Jeff Trawick <tr...@attglobal.net>.
Justin Erenkrantz <je...@apache.org> writes:
> On Tue, Apr 16, 2002 at 02:41:44PM -0400, Jeff Trawick wrote:
> > These facts are in conflict:
> >
> > 1) thread pool for worker threads is a subpool of pchild
> > 2) pchild can be cleaned up before worker threads go away
>
> Doesn't it seem that #2 is the real problem?
no way to make a thread disappear, no desire to bloat Apache and APR
with "should-I-go-away-yet?" logic all over the place
I don't see either of these as problems. They just are mutually
incompatible.
Something worker could do is create its own stand-alone pool and
allocate threads from that pool. That pool would never be cleaned
up.
> Why are we cleaning up pchild before its threads exit? Shouldn't
> we be doing a join on each thread - only after all threads exit
> do we exit the process and clean up pchild?
this particular scenario is a graceless shutdown of a particular
child...
> By "zapping pchild" do you mean calling apr_pool_clear(pchild) in
> clean_child_exit? -- justin
I mean clearing the pchild global variable.
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...
Re: Segfaults restarting on Linux with worker MPM
Posted by Jeff Trawick <tr...@attglobal.net>.
Aaron Bannert <aa...@clove.org> writes:
> On Tue, Apr 16, 2002 at 02:16:59PM -0700, Justin Erenkrantz wrote:
> > Why are we cleaning up pchild before its threads exit? Shouldn't
> > we be doing a join on each thread - only after all threads exit
> > do we exit the process and clean up pchild?
>
> Because right now there is no way to gracelessly kill a thread that is
> busy serving a long request.
>
> Does this mean that we need to check a "workers_may_exit"-like flag
> every time a syscall falls out with EINTR? This is kind of a problem,
> since we've decided to disallow pthread_kill-like functionality in
> APR (for good reason perhaps).
We'd do pthread_cancel() to catch the pthread calls which won't
return EINTR but are cancellation points, and we'd do pthread_kill()
to catch the non-pthread calls which return EINTR but aren't
cancellation points? But there are pthread functions (e.g.,
pthread_mutex_lock()) which won't return EINTR and are not cancellation
points. I guess at that point you assume that the thread owning the
mutex will hit a cancellation point or get the signal and then release
the mutex so that a thread blocked in pthread_mutex_lock() can reach
such a state. But for this you need to try the pthread_kill()
multiple times to make up for the fact that some will be eaten while
the thread is blocked in pthread_mutex_lock(). Or maybe the signal
handler sets a per-thread variable that Apache and APR can reference
:)
(I would interpret this as "I have no clue how to practically and
cleanly implement graceless termination.")
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...
Re: Segfaults restarting on Linux with worker MPM
Posted by Aaron Bannert <aa...@clove.org>.
On Tue, Apr 16, 2002 at 02:16:59PM -0700, Justin Erenkrantz wrote:
> Why are we cleaning up pchild before its threads exit? Shouldn't
> we be doing a join on each thread - only after all threads exit
> do we exit the process and clean up pchild?
Because right now there is no way to gracelessly kill a thread that is
busy serving a long request.
Does this mean that we need to check a "workers_may_exit"-like flag
every time a syscall falls out with EINTR? This is kind of a problem,
since we've decided to disallow pthread_kill-like functionality in
APR (for good reason perhaps).
-aaron
Re: Segfaults restarting on Linux with worker MPM
Posted by Justin Erenkrantz <je...@apache.org>.
On Tue, Apr 16, 2002 at 02:41:44PM -0400, Jeff Trawick wrote:
> These facts are in conflict:
>
> 1) thread pool for worker threads is a subpool of pchild
> 2) pchild can be cleaned up before worker threads go away
Doesn't it seem that #2 is the real problem?
Why are we cleaning up pchild before its threads exit? Shouldn't
we be doing a join on each thread - only after all threads exit
do we exit the process and clean up pchild?
By "zapping pchild" do you mean calling apr_pool_clear(pchild) in
clean_child_exit? -- justin
Re: Segfaults restarting on Linux with worker MPM
Posted by Justin Erenkrantz <je...@apache.org>.
On Tue, Apr 16, 2002 at 02:41:44PM -0400, Jeff Trawick wrote:
> Justin, if you are able to get coredumps for your problem, can you
> change clean_child_exit to zap pchild before doing the pool-destroy,
> and then see if pchild is already NULL when you segfault?
I will take a look at this once I get back from lunch. -- justin
Re: Segfaults restarting on Linux with worker MPM
Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick <tr...@attglobal.net> writes:
> Cliff Woolley <jw...@virginia.edu> writes:
>
> > On 15 Apr 2002, Jeff Trawick wrote:
> >
> > > But if I make a change to worker to get rid of the
> > > apr_bucket_alloc_destroy() I can't reproduce the problem.
> > >
> > > What is happening is that the thread's pool is getting cleaned up
> > > before the thread goes away (baaaad) since the thread pool is a
> > > subpool of pchild and pchild can go away before the thread with a
> > > graceful restart.
> >
> > Wow, that's funky. Sheesh.
>
> Actually the code doesn't bear that out for a graceful restart
> (sequence of cleaning up pchild vs. thread)... It is a non-graceful
> restart where child_main() exits without caring what the threads are
> doing. Back to the drawing board (this time with a debugger).
After another bit of debugging:
An easy way for me to recreate the segfault on AIX is to send SIGUSR1
followed by SIGTERM while the server is processing requests.
I changed clean_child_exit() to clear pchild before it does
apr_pool_destroy() on it. In cores for the apr_bucket_alloc_destroy()
segfault, pchild is already cleared, which means that cleanup on
pchild has already started.
Justin, if you are able to get coredumps for your problem, can you
change clean_child_exit to zap pchild before doing the pool-destroy,
and then see if pchild is already NULL when you segfault?
---
These facts are in conflict:
1) thread pool for worker threads is a subpool of pchild
2) pchild can be cleaned up before worker threads go away
Should APR require that the pool passed to apr_thread_create() live
until after threads exit, or should APR make the thread pool an
independent pool which won't be cleaned up with the pool passed to
apr_thread_create()?
The whole non-graceful termination scenario when there are multiple
threads seems ripe for problems. We want to be able to run cleanups
for pools, but we would like worker threads to simply vanish instead
of segfaulting because the main thread has already cleaned up some
resource. But threads don't vanish (portably).
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...
Re: Segfaults restarting on Linux with worker MPM
Posted by Jeff Trawick <tr...@attglobal.net>.
Cliff Woolley <jw...@virginia.edu> writes:
> On 15 Apr 2002, Jeff Trawick wrote:
>
> > But if I make a change to worker to get rid of the
> > apr_bucket_alloc_destroy() I can't reproduce the problem.
> >
> > What is happening is that the thread's pool is getting cleaned up
> > before the thread goes away (baaaad) since the thread pool is a
> > subpool of pchild and pchild can go away before the thread with a
> > graceful restart.
>
> Wow, that's funky. Sheesh.
Actually the code doesn't bear that out for a graceful restart
(sequence of cleaning up pchild vs. thread)... It is a non-graceful
restart where child_main() exits without caring what the threads are
doing. Back to the drawing board (this time with a debugger).
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...
Re: Segfaults restarting on Linux with worker MPM
Posted by Cliff Woolley <jw...@virginia.edu>.
On 15 Apr 2002, Jeff Trawick wrote:
> But if I make a change to worker to get rid of the
> apr_bucket_alloc_destroy() I can't reproduce the problem.
>
> What is happening is that the thread's pool is getting cleaned up
> before the thread goes away (baaaad) since the thread pool is a
> subpool of pchild and pchild can go away before the thread with a
> graceful restart.
Wow, that's funky. Sheesh.
> At the moment, the right fix seems to be for a thread pool to be
> stand-alone, but I'd guess there are at least 10 reasons I can't think
> of for why that is a stupid idea :)
I'll have to think on this for a while. My question is: why is pchild
getting cleaned up so early?
> Also, it would be nice if the apr_bucket_alloc code followed the usual
> pattern for destroy/cleanup (see apr_file_close() for an example,
> where we call apr_pool_cleanup_run()). It works as-is, but you have
> to stare at it for a while :)
Done. :)
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Segfaults restarting on Linux with worker MPM
Posted by Jeff Trawick <tr...@attglobal.net>.
Jeff Trawick <tr...@attglobal.net> writes:
> > Diff to previous 1.19 (colored)
> > apr_bucket_alloc_create() now registers a cleanup that does a
> > apr_bucket_alloc_destroy(). Don't double free by calling it explicitly.
> > ---------------------------------------------------------------
> >
> > In this case, it was that the apr_bucket_alloc_t was created in pconn.
> > The newly-added cleanup on pconn would destroy the apr_bucket_alloc_t.
> > Then an explicit call to apr_bucket_alloc_destroy *after* destroying pconn
> > would result in a double-free at best or an immediate segfault at worst.
>
> And I don't see that any change like is necessary on the worker MPM
> side.
But if I make a change to worker to get rid of the
apr_bucket_alloc_destroy() I can't reproduce the problem.
What is happening is that the thread's pool is getting cleaned up
before the thread goes away (baaaad) since the thread pool is a
subpool of pchild and pchild can go away before the thread with a
graceful restart. When this happens, the cleanup for the
bucket_alloc_t is run when pchild goes away, so by the time
apr_bucket_alloc_destroy() is called the storage has been released.
At the moment, the right fix seems to be for a thread pool to be
stand-alone, but I'd guess there are at least 10 reasons I can't think
of for why that is a stupid idea :)
Also, it would be nice if the apr_bucket_alloc code followed the usual
pattern for destroy/cleanup (see apr_file_close() for an example,
where we call apr_pool_cleanup_run()). It works as-is, but you have
to stare at it for a while :)
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...
Re: Segfaults restarting on Linux with worker MPM
Posted by Cliff Woolley <jw...@virginia.edu>.
On 15 Apr 2002, Jeff Trawick wrote:
> And I don't see that any change like is necessary on the worker MPM
> side.
I glanced at it briefly and was thinking the same thing, unfortunately.
Bah humbug. Back to the drawing board I go...
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Segfaults restarting on Linux with worker MPM
Posted by Jeff Trawick <tr...@attglobal.net>.
Cliff Woolley <jw...@virginia.edu> writes:
> On Mon, 15 Apr 2002, Justin Erenkrantz wrote:
>
> > I bet it could be related to a double-free condition. That would
> > make the most sense (to me).
> > I'm not sure which commit Cliff is talking about to OS/2.
>
> ---------------------------------------------------------------
> CVS log for httpd-2.0/server/mpm/mpmt_os2/mpmt_os2_child.c
>
> Revision 1.20 / (download) - annotate - [select for diffs] , Sun Apr 14
> 06:18:13 2002 UTC (35 hours, 43 minutes ago) by bjh
> Branch: MAIN
> CVS Tags: HEAD
> Changes since 1.19: +0 -1 lines
> Diff to previous 1.19 (colored)
> apr_bucket_alloc_create() now registers a cleanup that does a
> apr_bucket_alloc_destroy(). Don't double free by calling it explicitly.
> ---------------------------------------------------------------
>
> In this case, it was that the apr_bucket_alloc_t was created in pconn.
> The newly-added cleanup on pconn would destroy the apr_bucket_alloc_t.
> Then an explicit call to apr_bucket_alloc_destroy *after* destroying pconn
> would result in a double-free at best or an immediate segfault at worst.
And I don't see that any change like is necessary on the worker MPM
side.
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...
Re: Segfaults restarting on Linux with worker MPM
Posted by Cliff Woolley <jw...@virginia.edu>.
On Mon, 15 Apr 2002, Justin Erenkrantz wrote:
> I bet it could be related to a double-free condition. That would
> make the most sense (to me).
> I'm not sure which commit Cliff is talking about to OS/2.
---------------------------------------------------------------
CVS log for httpd-2.0/server/mpm/mpmt_os2/mpmt_os2_child.c
Revision 1.20 / (download) - annotate - [select for diffs] , Sun Apr 14
06:18:13 2002 UTC (35 hours, 43 minutes ago) by bjh
Branch: MAIN
CVS Tags: HEAD
Changes since 1.19: +0 -1 lines
Diff to previous 1.19 (colored)
apr_bucket_alloc_create() now registers a cleanup that does a
apr_bucket_alloc_destroy(). Don't double free by calling it explicitly.
---------------------------------------------------------------
In this case, it was that the apr_bucket_alloc_t was created in pconn.
The newly-added cleanup on pconn would destroy the apr_bucket_alloc_t.
Then an explicit call to apr_bucket_alloc_destroy *after* destroying pconn
would result in a double-free at best or an immediate segfault at worst.
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Segfaults restarting on Linux with worker MPM
Posted by Justin Erenkrantz <je...@apache.org>.
On Mon, Apr 15, 2002 at 07:17:39AM -0400, Jeff Trawick wrote:
> backtrace in that post. A rarer backtrace is where an internal free()
> routine segfaults (perhaps similar to your backtrace, where some heap
> structures are bogus, probably due to a bad free()?).
I bet it could be related to a double-free condition. That would
make the most sense (to me).
I'm not sure which commit Cliff is talking about to OS/2.
Cliff? -- justin
Re: Segfaults restarting on Linux with worker MPM
Posted by Cliff Woolley <jw...@virginia.edu>.
On 15 Apr 2002, Jeff Trawick wrote:
> See my post "watch out for new breakage" from last week. Almost
> immediately after a commit last week, I started getting segfaults
> consistently on AIX during a regression test run. I mentioned one
> backtrace in that post. A rarer backtrace is where an internal free()
> routine segfaults (perhaps similar to your backtrace, where some heap
> structures are bogus, probably due to a bad free()?).
I think Brian Havard figured this one out. I'll have to investigate and
see if his commit on the OS/2 side applies for the other MPMs as well, but
it could explain a lot.
--Cliff
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA
Re: Segfaults restarting on Linux with worker MPM
Posted by Jeff Trawick <tr...@attglobal.net>.
Justin Erenkrantz <je...@apache.org> writes:
> While hunting down some PRs, I find that I keep segfaulting with
> worker MPM on Linux 2.4 when I restart (sending it a SIGHUP).
See my post "watch out for new breakage" from last week. Almost
immediately after a commit last week, I started getting segfaults
consistently on AIX during a regression test run. I mentioned one
backtrace in that post. A rarer backtrace is where an internal free()
routine segfaults (perhaps similar to your backtrace, where some heap
structures are bogus, probably due to a bad free()?).
I haven't had a chance to debug. The regression test does initiate a
graceful restart right before most requests.
> (gdb) where
> #0 __pthread_mutex_lock (mutex=0x410) at mutex.c:99
> #1 0x40314098 in free () from /lib/libc.so.6
> #2 0x40045595 in apr_allocator_destroy (allocator=0x81d1210)
> at apr_pools.c:135
> #3 0x40018a55 in apr_bucket_alloc_destroy (list=0x81d1288)
> at apr_buckets_alloc.c:114
> #4 0x080b0177 in worker_thread (thd=0x8143270, dummy=0x819c398)
> at worker.c:877
> #5 0x4003f770 in dummy_worker (opaque=0x8143270) at thread.c:127
> #6 0x402880ae in pthread_start_thread (arg=0xbc3ffc00) at manager.c:291
>
> Can anyone else reproduce this? This is an SMP box, so maybe there
> is a race condition on shutdown? -- justin
I'm seeing it on a 1-way box and a 4-way box.
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...