You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Stas Bekman <st...@stason.org> on 2002/03/12 11:21:27 UTC
how to debug worker mpm httpd with gdb?
Any guidelines for debugging threaded (worker) mpm? I've read the
relevant gdb section, but it isn't enough. I couldn't find any relevant
info in httpd-docs-2.0/manual/developer.
I'm interested in tips making the debug process easier. Does anybody
have a cheatlist with useful tips/notes and willing to share it?
----
In particular how do I know which thread is going to process the next
request? If I start with -DONE_PROCESS and have:
<IfModule worker.c>
StartServers 1
MaxClients 1
MinSpareThreads 1
MaxSpareThreads 1
ThreadsPerChild 1
MaxRequestsPerChild 0
</IfModule>
httpd still keeps 4 threads, that's what gdb says:
(gdb) info thread
* 4 Thread 2051 (LWP 31642) apr_poll (aprset=0x81861f8,
nsds=0xbf5ffaec, timeout=-1) at poll.c:143
3 Thread 1026 (LWP 31641) 0x402267a2 in sigsuspend () from
/lib/libc.so.6
2 Thread 2049 (LWP 31640) 0x402dfd94 in poll () from /lib/libc.so.6
1 Thread 1024 (LWP 31592) 0x402267a2 in sigsuspend () from
/lib/libc.so.6
During the first request yet another thread starts. And one of the old
ones goes away.
It seems that the newest thread is the one I need to attach to. But when
I attach to it once, in a while I get switched to the previous thread,
etc. This is very confusing and non-productive :(
----
Another question is regarding:
set auto-solib-add 0
without this setting gdb tries to load/resolve symbols of many shared
libs and the startup may take 1 minute and more under mod_perl, which
sucks. With this setting I only need to load sharedlib which I need for
debugging, and the startup takes a few seconds.
The problem is that with worker-mpm and this setting in effect gdb won't
work properly. It doesn't see threads. 'info threads' returns nothing.
If I don't set auto-solib-add to 0, it works but takes ages to start.
Any ideas how to tell gdb not to resolve so's automatically and still
have threads working?
Thanks!
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://ticketmaster.com http://apacheweek.com
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: how to debug worker mpm httpd with gdb?
Posted by Jeff Trawick <tr...@attglobal.net>.
My 2 cents:
Keep using Linux when a fast build time is imperative or you're
debugging non-threaded apps.
Use AIX or Solaris with native tools when you want to debug threaded
apps.
It sucks :(
--
Jeff Trawick | trawick@attglobal.net
Born in Roswell... married an alien...
Re: how to debug worker mpm httpd with gdb?
Posted by Stas Bekman <st...@stason.org>.
I realize that the problem is in the debugger, not the MPM's cooperation
with it. If it was able to automatically switch to the thread when a
breakpoint is reached and stay there that's all I need and then I don't
care how many threads are running.
Moreover I definitely agree with Ryan, that in order to debug threads
you need more than one worker thread, otherwise you can do most of the
debugging with prefork mpm. Not that the potential problems will always
manifest them during the debugging, but it raises the chance that some
problems will be exposed early.
I guess my request was more of asking tips and tricks that I'm not aware
of to improve the threads debugging experience. I'm new to threads
(especially debugging threads), and I believe many others too. So having
a doc explaining debugging threads will save a lot of questions when
other module authors will start porting their modules to 2.0.
I volunteer to create such a doc based on what I learn, so if you can
help me that would be great!
Are there any books covering threads debugging in details? I've read
"Programming with POSIX(R) Threads" so far but it doesn't cover debugging.
> but I'd make sure you are running
> the latest gdb version (5.1.1?). I've found that the threading
> support was much improved from 4.x->5.x. -- justin
yes, I've the latest one.
Are there any other debuggers that are better than gdb concerning
threads, including commercial ones and running on Linux?
Thanks
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://ticketmaster.com http://apacheweek.com
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
Re: how to debug worker mpm httpd with gdb?
Posted by 'Justin Erenkrantz' <je...@ebuilt.com>.
On Tue, Mar 12, 2002 at 03:14:52PM -0800, Ryan Bloom wrote:
> Okay, but creating two threads won't solve the problem, and it may
> actually hinder people from using -X to debug thread related problems.
> The hardest part of debugging threaded code, is that the threads
> interact. By artificially limiting the number of threads, you are
> effectively removing the ability to debug thread interaction problems.
Correct which is the argument made why the MPM is doing the right
thing now.
My take is that there are two potential "debuggers" types - our
module writers and our MPM/thread experts. I'd prefer we default
to a mode that is beneficial to the module writers (one worker
thread only), and add another mode (say -DTHREAD_DEBUG - name sucks)
which allows the worker MPM to create its normal amount of threads
even though it is only using ONE_PROCESS. I think there will be
far more people attempting to debug their modules than our MPM.
> Also, on a platform like Linux, reducing to two threads really doesn't
> help anything, because the debuggers suck at dealing with threads.
Recent gdbs are much better at threads in Linux.
> What you really want is a good debugger, which will allow you to set a
> breakpoint, and the debugger will switch to the right thread when the
> breakpoint is hit.
Well, yeah, I'd like a nice debugger too. I'm a poor college
student, so I don't see that happening anytime soon unless gdb
gets an infusion of support. =) -- justin
Re: how to debug worker mpm httpd with gdb?
Posted by Aaron Bannert <aa...@clove.org>.
On Tue, Mar 12, 2002 at 04:26:19PM -0800, Justin Erenkrantz wrote:
> > Just change ThreadsPerChild to 1 if you want this. If you really want it,
> > I see no reason we can't create a -DONE_WORKER or similiar MPM-specific
> > flag to do the same.
>
> If you read Stas's message, he set TPC to 1, but it didn't work
> right. (The threads weren't peristent - they were always killed
> off.) -- justin
This sounds like a problem with gdb on linux w/ threads. I've set this
in solaris w/ gdb and not seen problems. I'd be surprised if threads
were being "killed off" after a request. (Perhaps Max and MinSpareThreads
shouldn't both be 1?)
I would expect there to be at least 3 threads when setting ThreadsPerChild
to 1:
worker thread
listener thread
POD thread/signal thread (signal thread is for -X)
Also, most thread libraries must create management threads for dealing
with LWP mapping, thread multiplexing, etc... which are also probably
showing up in the gdb output.
-aaron
Re: how to debug worker mpm httpd with gdb?
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Tue, Mar 12, 2002 at 04:23:47PM -0800, Aaron Bannert wrote:
> On Tue, Mar 12, 2002 at 03:37:20PM -0800, Justin Erenkrantz wrote:
> > -X should only create two threads: the manager thread and a single
> > worker thread. However, IIRC, it'll create ThreadsPerChild for that
> > one process. It should limit itself to just one worker thread.
> >
> > By limiting us to two httpd-created threads, it should be somewhat
> > easier for the debugger/user to determine which thread they should
> > attach to. IMHO, the current solution of creating one process with
> > a normal amount of workers isn't very nice. -- justin
>
> Just change ThreadsPerChild to 1 if you want this. If you really want it,
> I see no reason we can't create a -DONE_WORKER or similiar MPM-specific
> flag to do the same.
If you read Stas's message, he set TPC to 1, but it didn't work
right. (The threads weren't peristent - they were always killed
off.) -- justin
Re: how to debug worker mpm httpd with gdb?
Posted by Aaron Bannert <aa...@clove.org>.
On Tue, Mar 12, 2002 at 03:37:20PM -0800, Justin Erenkrantz wrote:
> -X should only create two threads: the manager thread and a single
> worker thread. However, IIRC, it'll create ThreadsPerChild for that
> one process. It should limit itself to just one worker thread.
>
> By limiting us to two httpd-created threads, it should be somewhat
> easier for the debugger/user to determine which thread they should
> attach to. IMHO, the current solution of creating one process with
> a normal amount of workers isn't very nice. -- justin
Just change ThreadsPerChild to 1 if you want this. If you really want it,
I see no reason we can't create a -DONE_WORKER or similiar MPM-specific
flag to do the same.
-aaron
RE: how to debug worker mpm httpd with gdb?
Posted by Ryan Bloom <rb...@covalent.net>.
> On Tue, Mar 12, 2002 at 03:06:13PM -0800, Ryan Bloom wrote:
> > How would you fix this? By having the worker MPM go into a single
> > threaded, single process mode with -X? That won't actually solve
the
> > problem, because the OS will still create the manager thread. This
is
> > just part of the pain of doing threaded debugging on Unix.
> >
>
> -X should only create two threads: the manager thread and a single
> worker thread. However, IIRC, it'll create ThreadsPerChild for that
> one process. It should limit itself to just one worker thread.
>
> By limiting us to two httpd-created threads, it should be somewhat
> easier for the debugger/user to determine which thread they should
> attach to. IMHO, the current solution of creating one process with
> a normal amount of workers isn't very nice. -- justin
Okay, but creating two threads won't solve the problem, and it may
actually hinder people from using -X to debug thread related problems.
The hardest part of debugging threaded code, is that the threads
interact. By artificially limiting the number of threads, you are
effectively removing the ability to debug thread interaction problems.
Also, on a platform like Linux, reducing to two threads really doesn't
help anything, because the debuggers suck at dealing with threads.
What you really want is a good debugger, which will allow you to set a
breakpoint, and the debugger will switch to the right thread when the
breakpoint is hit.
Ryan
Re: how to debug worker mpm httpd with gdb?
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Tue, Mar 12, 2002 at 03:06:13PM -0800, Ryan Bloom wrote:
> How would you fix this? By having the worker MPM go into a single
> threaded, single process mode with -X? That won't actually solve the
> problem, because the OS will still create the manager thread. This is
> just part of the pain of doing threaded debugging on Unix.
>
-X should only create two threads: the manager thread and a single
worker thread. However, IIRC, it'll create ThreadsPerChild for that
one process. It should limit itself to just one worker thread.
By limiting us to two httpd-created threads, it should be somewhat
easier for the debugger/user to determine which thread they should
attach to. IMHO, the current solution of creating one process with
a normal amount of workers isn't very nice. -- justin
RE: how to debug worker mpm httpd with gdb?
Posted by Ryan Bloom <rb...@covalent.net>.
> On Tue, Mar 12, 2002 at 06:21:27PM +0800, Stas Bekman wrote:
> > Any guidelines for debugging threaded (worker) mpm? I've read the
> > relevant gdb section, but it isn't enough. I couldn't find any
relevant
> > info in httpd-docs-2.0/manual/developer.
> >
> > I'm interested in tips making the debug process easier. Does anybody
> > have a cheatlist with useful tips/notes and willing to share it?
> <snip, snip>
> > During the first request yet another thread starts. And one of the
old
> > ones goes away.
> >
> > It seems that the newest thread is the one I need to attach to. But
when
> > I attach to it once, in a while I get switched to the previous
thread,
> > etc. This is very confusing and non-productive :(
>
> Yeah, well, I've maintained that worker doesn't have the right
> support for debugging. Nobody else seems to agree with me. =(
>
> AIUI (without placing words in anyone's mouth), it is that if you
> are using -X on worker, you want to debug the MPM *not* Apache.
> So, if you want to reliably test your code (i.e. a module), you
> should use the prefork MPM. If you are using worker, you need to
> attach to the "right" worker thread and hope you get the right
> thread. I've complained to the appropriate parties and they
> maintain their position is correct. Whatever. Feel free to
> join the complaint line. Worker is useless in -X mode.
How would you fix this? By having the worker MPM go into a single
threaded, single process mode with -X? That won't actually solve the
problem, because the OS will still create the manager thread. This is
just part of the pain of doing threaded debugging on Unix.
Ryan
Re: how to debug worker mpm httpd with gdb?
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Tue, Mar 12, 2002 at 06:21:27PM +0800, Stas Bekman wrote:
> Any guidelines for debugging threaded (worker) mpm? I've read the
> relevant gdb section, but it isn't enough. I couldn't find any relevant
> info in httpd-docs-2.0/manual/developer.
>
> I'm interested in tips making the debug process easier. Does anybody
> have a cheatlist with useful tips/notes and willing to share it?
<snip, snip>
> During the first request yet another thread starts. And one of the old
> ones goes away.
>
> It seems that the newest thread is the one I need to attach to. But when
> I attach to it once, in a while I get switched to the previous thread,
> etc. This is very confusing and non-productive :(
Yeah, well, I've maintained that worker doesn't have the right
support for debugging. Nobody else seems to agree with me. =(
AIUI (without placing words in anyone's mouth), it is that if you
are using -X on worker, you want to debug the MPM *not* Apache.
So, if you want to reliably test your code (i.e. a module), you
should use the prefork MPM. If you are using worker, you need to
attach to the "right" worker thread and hope you get the right
thread. I've complained to the appropriate parties and they
maintain their position is correct. Whatever. Feel free to
join the complaint line. Worker is useless in -X mode.
> Any ideas how to tell gdb not to resolve so's automatically and still
> have threads working?
Don't have an idea about this, but I'd make sure you are running
the latest gdb version (5.1.1?). I've found that the threading
support was much improved from 4.x->5.x. -- justin