You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by "Jani M." <mo...@iki.fi> on 2007/06/03 21:14:55 UTC
[mp2] Segmentation faults with threaded worker-mpm
Hi all,
I have a problem with segmentation faults which I've been unsuccessful
at solving so far. These only occure when running the handler+filter in
a threaded worker-mpm setup, a preforked environment does not have any
problems.
First, the environment. I am running on Debian Etch, with Apache 2.2.3,
Perl 5.8.8 and the latest (on 31.5.) mod_perl from trunk with the two
patches posted by Torsten to mod-perl-dev on May.
The mod_perl handler/filter is fairly complex, consisting of:
- a PerlPostReadRequestHandler
- a PerlFixupHandler
- and a HTTP request inputfilter
The setup implements:
- on-the-fly mod_proxy configuration to select the correct backend
server based on the client request (PostReadRequestHandler + FixupHandler)
- POST to GET conversion for some requests (PostReadRequestHandler)
- POST request pass-through for others (InputFilter to re-post body)
- Regular GET request parsing and validation (PostReadRequestHandler)
- Backend-server response caching by mod_cache (for GET requests)
As said, when running with a preforked mpm, everything works great - no
segfaults at all. Also, when running on a worker mpm setup with only
*one* perl interpreter per process, the setup also works without any
problems. The problems start only when two or more interpreters per
process is running - unfortunately, this is exactly what I would need
for scalability and performance.
In addition to the segfaults (and quite often a bit prior to them), I
see one or more of the following type of error in the apache error log:
[error] lookup of 'MyApache::MyModule::' failed
[error] lookup of 'MyApache::::' failed
[error] lookup of 'MyApache::MyM\xf6\xe1\xa1le::rewrite_handler' failed
[error] lookup of '\x88\x8a\xa5\b\x10' failed
The above are examples only, and the "corruption" in the strings is
random. These errors may also appear with no segfault following them. No
other suspicious erros are visible in the logs.
The setup handles a large volume of requests - anywhere from 10-20
requests/s upto 100-200 requests/s at peaks. With these volumes, I
usually see a segmentation fault on average every couple minutes.
MaxRequestPerChild is set to 5000. Lowering this will make the faults a
bit less infrequent, but they are there still none the less.
PerlInterpMaxRequests is currently at 500 - lowering could also make the
segfaults less infrequent, but would not remove them entirely.
I've done my best to ensure that all the perl code is thread safe, and
cannot find any problems with it myself. The PostReadRequestHandler uses
sockets to communicate with a seperate process, but I've tested the code
with all socket communications removed, and it still segfaulted. The
rest of the code is pretty much standard and straighforward processing
of the request, and I cannot see how that could cause problems with
threading - only functions and methods provided by Apache2:: or APR::
are used.
The code does use the $r->notes quite a bit to store content between the
handlers and filters, and I've somehow been suspecting that maybe the
memory corruption (or what atleast looks like that) comes from somewhere
over there..
Any thoughts, ideas, solutions or further clarifying questions on the
subject are welcome!
--
Jani
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by "Jani M." <mo...@iki.fi>.
Torsten Foertsch wrote:
> Try $r->push_handlers(PerlFixupHandler => 'My::fixup_handler').
Doesn't make any difference unfortunately. Strange though, when I use
the hard reference \&fixup_handler, the lookup-errors (which are
sometimes corrupted) in the log still display as My::fixup_handler.
Cheers,
-- Jani
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Torsten Foertsch <to...@gmx.net>.
On Tuesday 12 June 2007 17:40, Jani M. wrote:
> After a few moments of trial and error, I tracked the problem down to
> $r->push_handlers(PerlFixupHandler => \&fixup_handler). The segfaults
> start even if this is the only thing the original postreadrequesthandler
> does, and even if the fixuphandler does absolutely nothing. If I set the
> same fixuphandler in the server config file, and don't use
> push_handlers, the problem goes away.
$r->push_handlers(PerlFixupHandler => \&fixup_handler) is different from
"PerlFixupHandler My::fixup_handler". The former will probably not work
with "PerlInterpScope handler" because \&fixup_handler is a hard reference
valid only in one interpreter. Normally this would cause a "Cannot resolve
handler" or so message.
Try $r->push_handlers(PerlFixupHandler => 'My::fixup_handler').
Torsten
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Fred Moyer <fr...@redhotpenguin.com>.
Jani M. wrote:
> Jani M. wrote:
> After a few moments of trial and error, I tracked the problem down to
> $r->push_handlers(PerlFixupHandler => \&fixup_handler). The segfaults
> start even if this is the only thing the original postreadrequesthandler
> does, and even if the fixuphandler does absolutely nothing. If I set the
> same fixuphandler in the server config file, and don't use
> push_handlers, the problem goes away.
So you push this fixup handler in the postreadrequesthandler? There is
a known issue with this which sounds like it could be related:
http://marc.info/?l=apache-modperl&m=115812572132632&w=2
I have dug into the issue somewhat but have been caught up in a tuit
drought lately and haven't made much progress on it. The gist of it is,
if you push a handler onto a different phase, any existing handlers for
that phase are removed and replaced with the new pushed handler. I
haven't tried this under threads but have been able to replicate the
issue in a reproducible test case (see mail thread).
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by "Jani M." <mo...@iki.fi>.
Jani M. wrote:
> I've still been trying to identify the root cause of the crashes, but
> unfortunately haven't had success yet.
Looks like success at last. I managed to tune my test environment server
load from <30 requests/s to 500+ requests/s. This in turn started
producing segfaults in the test environment too.
After a few moments of trial and error, I tracked the problem down to
$r->push_handlers(PerlFixupHandler => \&fixup_handler). The segfaults
start even if this is the only thing the original postreadrequesthandler
does, and even if the fixuphandler does absolutely nothing. If I set the
same fixuphandler in the server config file, and don't use
push_handlers, the problem goes away.
As I've applied Torsten's push_handler-patch from last month, this is
probably a different, although possibly related problem. The exact
nature of the problem is beyond my skills, but hopefully this will help
to track down the problem.
Cheers,
-- Jani
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by "Jani M." <mo...@iki.fi>.
Jani M. wrote:
> As far as I have seen, all the segfaults seem to be mgv related, but not
> at the exact same spot.
I've still been trying to identify the root cause of the crashes, but
unfortunately haven't had success yet. Some of the things I've tried so far:
- Different PerlInterpScope settings
- Disabling KeepAlive connections
- Switching Perl code where possible to run at different handler phases
- Disabling mod_cache
- and many other things I can't even remember anymore :)
As I have the setup running on several servers, I've been able to
compare how the segfaults relate to server load. This is where things
start to get more interesting.
On a server handling somewhere around 30 requests/s during peaks, I
don't see almost any segfaults at all. Maybe one or two max per day,
sometimes none. However, a server doing around 200 requests/s during the
daytime crashes a lot more frequently - I've seen upto 150 segfaults per
day, with the average being somewhere around 100.
To me atleast this looks like classic signs of problems related to race
conditions. As the server load increases, so does the likelyhood of
triggering such an event. This probably also explains why I can't
reproduce the segfaults in my test environment - I don't have enough
power there to generate these levels of requests/s. Does this sound like
a feasible conclusion?
All tips in debugging this further are welcome - I'm really not that
familiar with threaded coding.
> One odd thing I noticed is that I have configured mod_perl to run 4
> interpreters per process, yet I always see only 2 perl threads in the
> core dumps. Is there a reason for this?
This is still bothering me - does anyone have any theories yet as to why
this is?
Cheers,
-- Jani
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by "Jani M." <mo...@iki.fi>.
> Try "PerlInterpScope request" and see if the coredumps disappear. Further
> check if it's always this place.
PerlInterpScope request did not help. I just tested, and still got a
segfault.
As far as I have seen, all the segfaults seem to be mgv related, but not
at the exact same spot.
One odd thing I noticed is that I have configured mod_perl to run 4
interpreters per process, yet I always see only 2 perl threads in the
core dumps. Is there a reason for this?
Since it looked like all the problems always occured in the FixupHandler
phase, I tried moving the code I have there to HeaderParser phase, but
that did not help. Here is an example backtrace from that test. Only one
of the perl threads was in the HeaderParser phase this time - the other
thread was polling for data from the client.
[Switching to thread 12 (process 15427)]#0 0xb7bc0d51 in kill () from
/lib/tls/libc.so.6
#0 0xb7bc0d51 in kill () from /lib/tls/libc.so.6
#1 0x0807c9bb in ap_fatal_signal_child_setup ()
#2 <signal handler called>
#3 0xb781149d in modperl_mgv_lookup (my_perl=0x8677208,
symbol=0x812f8c0) at modperl_mgv.c:131
#4 0xb7811562 in modperl_mgv_lookup_autoload (my_perl=0x8677208,
symbol=0x812f8c0, s=0x80a88c8, p=0x8937320) at modperl_mgv.c:153
#5 0xb78060f1 in modperl_callback (my_perl=0x8677208,
handler=0x8939100, p=0x8937320, r=0x8937358, s=0x80a88c8, args=0x87cca2c)
at modperl_callback.c:75
#6 0xb7806b79 in modperl_callback_run_handlers (idx=0, type=4,
r=0x8937358, c=0x0, s=0x80a88c8, pconf=0x0, plog=0x0, ptemp=0x0,
run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:263
#7 0xb78070aa in modperl_callback_per_dir (idx=0, r=0x8937358,
run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:370
#8 0xb7825a64 in modperl_header_parser_handler (r=0x8937358) at
modperl_hooks.c:32
#9 0x080743f9 in ap_run_header_parser ()
#10 0x08071eb1 in ap_process_request_internal ()
#11 0x08084528 in ap_process_request ()
#12 0x080817de in ap_register_input_filter ()
#13 0x0807b507 in ap_run_process_connection ()
#14 0x0808914b in ap_graceful_stop_signalled ()
#15 0xb7d37316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#16 0xb7cd40bd in start_thread () from /lib/tls/libpthread.so.0
#17 0xb7c639ee in clone () from /lib/tls/libc.so.6
And here are is one more example when running in the FixupHandler phase,
again crashing at a slightly different location.
[Switching to thread 7 (process 11705)]#0 0xb7bb1d51 in kill () from
/lib/tls/libc.so.6
#0 0xb7bb1d51 in kill () from /lib/tls/libc.so.6
#1 0x0807c9bb in ap_fatal_signal_child_setup ()
#2 <signal handler called>
#3 0xb78022d8 in modperl_mgv_compile (my_perl=0x81cf6c0, p=0x80a20d0,
name=0x878bc62 "MyModule") at modperl_mgv.c:98
#4 0xb7802bd1 in modperl_mgv_resolve (my_perl=0x81cf6c0,
handler=0x88bdaf0, p=0x80a20d0,
name=0x88bdac0 "MyApache::MyModule::proxy_rewrite_handler",
logfailure=0) at modperl_mgv.c:275
#5 0xb77f8cb0 in modperl_handler_resolve (my_perl=0x81cf6c0,
handp=0xb451d274, p=0x88bc3d0, s=0x80a88c8) at modperl_handler.c:233
#6 0xb77f6e73 in modperl_callback (my_perl=0x81cf6c0,
handler=0x88bdaf0, p=0x88bc3d0, r=0x88bc408, s=0x80a88c8, args=0x88c5ea0)
at modperl_callback.c:39
#7 0xb77f7b79 in modperl_callback_run_handlers (idx=5, type=4,
r=0x88bc408, c=0x0, s=0x80a88c8, pconf=0x0, plog=0x0, ptemp=0x0,
run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:263
#8 0xb77f80aa in modperl_callback_per_dir (idx=5, r=0x88bc408,
run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:370
#9 0xb7816b63 in modperl_fixup_handler (r=0x88bc408) at modperl_hooks.c:57
#10 0x0806ff29 in ap_run_fixups ()
#11 0x08084528 in ap_process_request ()
#12 0x080817de in ap_register_input_filter ()
#13 0x0807b507 in ap_run_process_connection ()
#14 0x0808914b in ap_graceful_stop_signalled ()
#15 0xb7d28316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#16 0xb7cc50bd in start_thread () from /lib/tls/libpthread.so.0
#17 0xb7c549ee in clone () from /lib/tls/libc.so.6
--
Jani
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Torsten Foertsch <to...@gmx.net>.
On Wednesday 06 June 2007 10:55, Jani M. wrote:
> (gdb) btt 4
> [Switching to thread 4 (process 2133)]#0 0xb77eb15a in
> modperl_mgv_as_string (my_perl=0x8662c58, symbol=0x8178190, p=0x8938438,
> package=0) at modperl_mgv.c:399
> 399 modperl_mgv.c: No such file or directory.
> in modperl_mgv.c
> #0 0xb77eb15a in modperl_mgv_as_string (my_perl=0x8662c58,
> symbol=0x8178190, p=0x8938438, package=0) at modperl_mgv.c:399
> #1 0xb77df146 in modperl_callback (my_perl=0x8662c58,
> handler=0x8939ee8, p=0x8938438, r=0x8938470, s=0x80a88c8, args=0x870c29c)
> at modperl_callback.c:85
> #2 0xb77dfb79 in modperl_callback_run_handlers (idx=5, type=4,
> r=0x8938470, c=0x0, s=0x80a88c8, pconf=0x0, plog=0x0, ptemp=0x0,
> run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:263
> #3 0xb77e00aa in modperl_callback_per_dir (idx=5, r=0x8938470,
> run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:370
> #4 0xb77feb63 in modperl_fixup_handler (r=0x8938470) at modperl_hooks.c:57
> #5 0x0806ff29 in ap_run_fixups ()
> #6 0x08084528 in ap_process_request ()
> #7 0x080817de in ap_register_input_filter ()
> #8 0x0807b507 in ap_run_process_connection ()
> #9 0x0808914b in ap_graceful_stop_signalled ()
> #10 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
> #11 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
> #12 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
...
> (gdb) btt 9
> [Switching to thread 9 (process 2125)]#0 0xb7b99d51 in kill () from
> /lib/tls/libc.so.6
> #0 0xb7b99d51 in kill () from /lib/tls/libc.so.6
> #1 0x0807c9bb in ap_fatal_signal_child_setup ()
> #2 <signal handler called>
> #3 0xb77eb15a in modperl_mgv_as_string (my_perl=0x85bc758,
> symbol=0x8178190, p=0x88ad7f8, package=0) at modperl_mgv.c:399
> #4 0xb77df146 in modperl_callback (my_perl=0x85bc758,
> handler=0x88af2a8, p=0x88ad7f8, r=0x88ad830, s=0x80a88c8, args=0x86eaf0c)
> at modperl_callback.c:85
> #5 0xb77dfb79 in modperl_callback_run_handlers (idx=5, type=4,
> r=0x88ad830, c=0x0, s=0x80a88c8, pconf=0x0, plog=0x0, ptemp=0x0,
> run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:263
> #6 0xb77e00aa in modperl_callback_per_dir (idx=5, r=0x88ad830,
> run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:370
> #7 0xb77feb63 in modperl_fixup_handler (r=0x88ad830) at modperl_hooks.c:57
> #8 0x0806ff29 in ap_run_fixups ()
> #9 0x08084528 in ap_process_request ()
> #10 0x080817de in ap_register_input_filter ()
> #11 0x0807b507 in ap_run_process_connection ()
> #12 0x0808914b in ap_graceful_stop_signalled ()
> #13 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
> #14 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
> #15 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
Interesting both threads are at exactly the same line of code:
ptr = string = apr_palloc(p, len+1);
for (mgv = symbol; (package ? mgv->next : mgv); mgv = mgv->next) {
==> Copy(mgv->name, ptr, mgv->len, char);
ptr += mgv->len;
}
I suspect mgv has been corrupted. The question is who is the culprit?
Try "PerlInterpScope request" and see if the coredumps disappear. Further
check if it's always this place.
Torsten
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Perrin Harkins <pe...@elem.com>.
On 6/4/07, Issac Goldstand <ma...@beamartyr.net> wrote:
> I'm a bit confused here... Perrin, isn't what Jani is mentioning here
> exactly what Stas wanted to accomplish (well, one specific detail of
> what he wanted to accomplish) with mp2, with the specific result in mind
> of eliminating the common Apache 1 issue of using the 2 backend
> (mp/static) with a single front-end, or am I dreaming?
That was discussed as one possible benefit. However, the nature of
perl threads has kind of killed that idea. They ended up using a lot
more memory than forking because of copy-on-write. This is why
prefork is the recommended way to run mod_perl on all platforms other
than Windows.
- Perrin
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Issac Goldstand <ma...@beamartyr.net>.
I'm a bit confused here... Perrin, isn't what Jani is mentioning here
exactly what Stas wanted to accomplish (well, one specific detail of
what he wanted to accomplish) with mp2, with the specific result in mind
of eliminating the common Apache 1 issue of using the 2 backend
(mp/static) with a single front-end, or am I dreaming?
Issac
Perrin Harkins wrote:
> On 6/4/07, Jani M. <mo...@iki.fi> wrote:
>> With prefork, running ~ 600 processes results in roughly 700MB of memory
>> consumption. This obviously gives us 600 client "download slots", and
>> 600 perl interpreters, and leaves some memory for other processes and
>> disk caching.
>>
>> With the threaded worker mpm, I can run ~150 processes, each with 20
>> threads, and 3 perl interpreters. This will use about the same about of
>> memory, so roughly 700MB. This in turn gives us 3000 client "download
>> slots", but "only" 450 perl interpreters.
>
> There's no reason to run more interpreters under prefork. Don't make
> your mod_perl server handle static files. Serve them with a separate
> apache or some other proxy server.
>
> But I think the bigger picture that you're missing here is
> copy-on-write sharing. When you fork processes, most of the memory is
> actually shared internally by the virtual memory system, so it isn't
> using physical RAM. You might have very large processes which only
> use a few MBs of actual RAM each. Threads don't get this benefit.
> They have to copy everything when a new thread is started and there is
> no copy-on-write sharing. It results in some really large differences
> in terms of physical RAM needed per perl interpreter. If you run 100
> interpreters with a threaded MPM and 100 with prefork, the prefork
> ones will use a lot less of your real memory.
>
>> When most of the time the client uses is not spent inside the mod_perl
>> handlers or filters, but rather downloading the actual content,
>> 'PerlInterpScope handler' can be a lifesaver - a few interpreters can
>> easily handle the load for a much higher number of client threads.
>
> That's what the proxy setup is for. The mod_perl setup just dumps the
> file very quickly to the proxy, and the proxy deals with spooning it
> out the clients.
>
>> I know what you mean, but the problem here is that this mod_perl server
>> *is* the reverse proxy :) There are several backend servers which this
>> server will both proxy and cache the content for - mod_perl is, putting
>> it simply, just needed for additional intelligence.
>
> If what you're saying is that you can't separate out the mod_perl bits
> with a proxy because they do things like authentication, you might be
> interested in seeing what LiveJournal does with their proxy called
> perlbal. They use a system of internal redirects to let mod_perl
> handle auth functions and pass the file serving off to perlbal. You
> can read about it in Brad Fitzpatrick's presentation (around silde
> 45):
> http://www.danga.com/words/2007_04_linuxfest_nw/linuxfest.pdf
>
> - Perrin
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Perrin Harkins <pe...@elem.com>.
On 6/4/07, Jani M. <mo...@iki.fi> wrote:
> That's unfortunately exactly the problem. It's not authentication, but
> the mod_perl bits are still required before proxying can be done.
The scenario I was referring to is where your front-end proxy is not
running mod_perl, and it just sends the request to a backend mod_perl
server. If that mod_perl server does some work and proxies it again
to another server, it will still work. The mod_perl server will push
the file to the frontend proxy very quickly, and move on to the next
request while the proxy waits for the client to finish.
- Perrin
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by "Jani M." <mo...@iki.fi>.
Perrin Harkins wrote:
> If what you're saying is that you can't separate out the mod_perl bits
> with a proxy because they do things like authentication, you might be
That's unfortunately exactly the problem. It's not authentication, but
the mod_perl bits are still required before proxying can be done.
> interested in seeing what LiveJournal does with their proxy called
> perlbal. They use a system of internal redirects to let mod_perl
> handle auth functions and pass the file serving off to perlbal. You
> can read about it in Brad Fitzpatrick's presentation (around silde
> 45):
> http://www.danga.com/words/2007_04_linuxfest_nw/linuxfest.pdf
Very interesting reading, thanks for the pointer. I will certainly take
a closer look at Perlbal.
Cheers,
-- Jani
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Frank Wiles <fr...@wiles.org>.
On Mon, 4 Jun 2007 14:30:01 -0400
"Perrin Harkins" <pe...@elem.com> wrote:
> > I know what you mean, but the problem here is that this mod_perl
> > server *is* the reverse proxy :) There are several backend servers
> > which this server will both proxy and cache the content for -
> > mod_perl is, putting it simply, just needed for additional
> > intelligence.
>
> If what you're saying is that you can't separate out the mod_perl bits
> with a proxy because they do things like authentication, you might be
> interested in seeing what LiveJournal does with their proxy called
> perlbal. They use a system of internal redirects to let mod_perl
> handle auth functions and pass the file serving off to perlbal. You
> can read about it in Brad Fitzpatrick's presentation (around silde
> 45):
> http://www.danga.com/words/2007_04_linuxfest_nw/linuxfest.pdf
>
> - Perrin
I second that recommendation. Perlbal rocks, I use it on nearly
every project I do these days.
---------------------------------
Frank Wiles <fr...@wiles.org>
http://www.wiles.org
---------------------------------
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Perrin Harkins <pe...@elem.com>.
On 6/4/07, Jani M. <mo...@iki.fi> wrote:
> With prefork, running ~ 600 processes results in roughly 700MB of memory
> consumption. This obviously gives us 600 client "download slots", and
> 600 perl interpreters, and leaves some memory for other processes and
> disk caching.
>
> With the threaded worker mpm, I can run ~150 processes, each with 20
> threads, and 3 perl interpreters. This will use about the same about of
> memory, so roughly 700MB. This in turn gives us 3000 client "download
> slots", but "only" 450 perl interpreters.
There's no reason to run more interpreters under prefork. Don't make
your mod_perl server handle static files. Serve them with a separate
apache or some other proxy server.
But I think the bigger picture that you're missing here is
copy-on-write sharing. When you fork processes, most of the memory is
actually shared internally by the virtual memory system, so it isn't
using physical RAM. You might have very large processes which only
use a few MBs of actual RAM each. Threads don't get this benefit.
They have to copy everything when a new thread is started and there is
no copy-on-write sharing. It results in some really large differences
in terms of physical RAM needed per perl interpreter. If you run 100
interpreters with a threaded MPM and 100 with prefork, the prefork
ones will use a lot less of your real memory.
> When most of the time the client uses is not spent inside the mod_perl
> handlers or filters, but rather downloading the actual content,
> 'PerlInterpScope handler' can be a lifesaver - a few interpreters can
> easily handle the load for a much higher number of client threads.
That's what the proxy setup is for. The mod_perl setup just dumps the
file very quickly to the proxy, and the proxy deals with spooning it
out the clients.
> I know what you mean, but the problem here is that this mod_perl server
> *is* the reverse proxy :) There are several backend servers which this
> server will both proxy and cache the content for - mod_perl is, putting
> it simply, just needed for additional intelligence.
If what you're saying is that you can't separate out the mod_perl bits
with a proxy because they do things like authentication, you might be
interested in seeing what LiveJournal does with their proxy called
perlbal. They use a system of internal redirects to let mod_perl
handle auth functions and pass the file serving off to perlbal. You
can read about it in Brad Fitzpatrick's presentation (around silde
45):
http://www.danga.com/words/2007_04_linuxfest_nw/linuxfest.pdf
- Perrin
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by "Jani M." <mo...@iki.fi>.
Perrin Harkins wrote:
> No, it's just the opposite. Using prefork won't save CPU, but it will
> save memory, meaning you can run more perl interpreters.
I'm sorry, but I have to disagree here - especially with the
configuration I need. Let's take the server with 1 GB of memory as an
example.
With prefork, running ~ 600 processes results in roughly 700MB of memory
consumption. This obviously gives us 600 client "download slots", and
600 perl interpreters, and leaves some memory for other processes and
disk caching.
With the threaded worker mpm, I can run ~150 processes, each with 20
threads, and 3 perl interpreters. This will use about the same about of
memory, so roughly 700MB. This in turn gives us 3000 client "download
slots", but "only" 450 perl interpreters.
When most of the time the client uses is not spent inside the mod_perl
handlers or filters, but rather downloading the actual content,
'PerlInterpScope handler' can be a lifesaver - a few interpreters can
easily handle the load for a much higher number of client threads.
> The normal way to do this is to have your mod_perl server separate
> from your static file server, usually by doing a reverse proxy to the
> mod_perl server. Then the static server handles doling out bytes to
> slow clients. A few variations of this setup are discussed in the
> docs.
I know what you mean, but the problem here is that this mod_perl server
*is* the reverse proxy :) There are several backend servers which this
server will both proxy and cache the content for - mod_perl is, putting
it simply, just needed for additional intelligence.
> Honestly, the person who has done the most work on debugging thread
> crashes is Torsten. His advice on how to debug it will be better than
> mine. It does seem like people usually solve them by using backtrace
> analysis though.
Torsten suggested that I'd take a core dump and try to see which memory
regions are corrupted. As working with dumps, debuggers and the whole
lot is definately not one of my strongest suits, all help here would be
most appreciated. I think I could manage to actually get my hands on a
dump, but what to do after that is a bit of a mystery..
Cheers,
-- Jani
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by "Jani M." <mo...@iki.fi>.
Perrin Harkins wrote:
> Honestly, the person who has done the most work on debugging thread
> crashes is Torsten. His advice on how to debug it will be better than
> mine. It does seem like people usually solve them by using backtrace
> analysis though.
Getting back to this, I've now had time to install debug versions of
mod_perl module and perl to the production server, and run backtraces on
a core dump. Unfortunately I do not have the possibility to run a debug
version of Apache right now, but hopefully that won't be needed.
Looking at the backtraces for each thread, there are a couple things
that I think I can see:
- The segfault appears to happen when Apache attempts to recycle the
child process (ap_graceful_stop_signalled in thread #1)
- Threads 4 and 9 appear to be running perl, and those are also the only
threads that show anything interesting (atleast to me)
As per the instructions on mod_perl documention, I analyzed the perl
threads a bit more closely. The 'curinfo' returns for both threads
'536870923Cannot access memory at address 0x4040004'.
If anyone has any ideas/thoughts on how to further debug the problem
based on the backtraces, please let me know.
Backtraces:
(gdb) btt 1
[Switching to thread 1 (process 2094)]#0 0xb7cb2401 in __read_nocancel
() from /lib/tls/libpthread.so.0
#0 0xb7cb2401 in __read_nocancel () from /lib/tls/libpthread.so.0
#1 0x0808b855 in ap_mpm_pod_check ()
#2 0x080894d5 in ap_graceful_stop_signalled ()
#3 0x08089656 in ap_graceful_stop_signalled ()
#4 0x08089715 in ap_graceful_stop_signalled ()
#5 0x0808a807 in ap_mpm_run ()
#6 0x0806232f in main ()
(gdb) btt 2
[Switching to thread 2 (process 2135)]#0 0xb7cb25fe in accept () from
/lib/tls/libpthread.so.0
#0 0xb7cb25fe in accept () from /lib/tls/libpthread.so.0
#1 0xb7d0c5bd in apr_socket_accept () from /usr/lib/libapr-1.so.0
#2 0x0808b98c in unixd_accept ()
#3 0x08088b09 in ap_graceful_stop_signalled ()
#4 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#5 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
#6 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
(gdb) btt 3
[Switching to thread 3 (process 2134)]#0 0xb7c32ef9 in poll () from
/lib/tls/libc.so.6
#0 0xb7c32ef9 in poll () from /lib/tls/libc.so.6
#1 0xb7d10145 in apr_wait_for_io_or_timeout () from /usr/lib/libapr-1.so.0
#2 0xb7d0b6b3 in apr_socket_recv () from /usr/lib/libapr-1.so.0
#3 0x0807b756 in ap_lingering_close ()
#4 0x08089153 in ap_graceful_stop_signalled ()
#5 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#6 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
#7 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
(gdb) btt 4
[Switching to thread 4 (process 2133)]#0 0xb77eb15a in
modperl_mgv_as_string (my_perl=0x8662c58, symbol=0x8178190, p=0x8938438,
package=0) at modperl_mgv.c:399
399 modperl_mgv.c: No such file or directory.
in modperl_mgv.c
#0 0xb77eb15a in modperl_mgv_as_string (my_perl=0x8662c58,
symbol=0x8178190, p=0x8938438, package=0) at modperl_mgv.c:399
#1 0xb77df146 in modperl_callback (my_perl=0x8662c58,
handler=0x8939ee8, p=0x8938438, r=0x8938470, s=0x80a88c8, args=0x870c29c)
at modperl_callback.c:85
#2 0xb77dfb79 in modperl_callback_run_handlers (idx=5, type=4,
r=0x8938470, c=0x0, s=0x80a88c8, pconf=0x0, plog=0x0, ptemp=0x0,
run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:263
#3 0xb77e00aa in modperl_callback_per_dir (idx=5, r=0x8938470,
run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:370
#4 0xb77feb63 in modperl_fixup_handler (r=0x8938470) at modperl_hooks.c:57
#5 0x0806ff29 in ap_run_fixups ()
#6 0x08084528 in ap_process_request ()
#7 0x080817de in ap_register_input_filter ()
#8 0x0807b507 in ap_run_process_connection ()
#9 0x0808914b in ap_graceful_stop_signalled ()
#10 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#11 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
#12 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
(gdb) btt 5
[Switching to thread 5 (process 2132)]#0 0xb7cafc01 in
pthread_cond_wait@@GLIBC_2.3.2 () from /lib/tls/libpthread.so.0
#0 0xb7cafc01 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/tls/libpthread.so.0
#1 0xb7d053da in apr_thread_cond_wait () from /usr/lib/libapr-1.so.0
#2 0x0808b2b3 in ap_queue_pop ()
#3 0x08088fc5 in ap_graceful_stop_signalled ()
#4 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#5 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
#6 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
(gdb) btt 6
[Switching to thread 6 (process 2131)]#0 0xb7c32ef9 in poll () from
/lib/tls/libc.so.6
#0 0xb7c32ef9 in poll () from /lib/tls/libc.so.6
#1 0xb7d10145 in apr_wait_for_io_or_timeout () from /usr/lib/libapr-1.so.0
#2 0xb7d0b6b3 in apr_socket_recv () from /usr/lib/libapr-1.so.0
#3 0x0807b756 in ap_lingering_close ()
#4 0x08089153 in ap_graceful_stop_signalled ()
#5 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#6 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
#7 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
(gdb) btt 7
[Switching to thread 7 (process 2130)]#0 0xb7c32ef9 in poll () from
/lib/tls/libc.so.6
#0 0xb7c32ef9 in poll () from /lib/tls/libc.so.6
#1 0xb7d10145 in apr_wait_for_io_or_timeout () from /usr/lib/libapr-1.so.0
#2 0xb7d0b6b3 in apr_socket_recv () from /usr/lib/libapr-1.so.0
#3 0xb7ef1967 in apr_bucket_socket_create () from
/usr/lib/libaprutil-1.so.0
#4 0xb7ef357a in apr_brigade_split_line () from /usr/lib/libaprutil-1.so.0
#5 0x0807407b in ap_core_input_filter ()
#6 0x080810ce in ap_register_input_filter ()
#7 0x08068c2e in ap_rgetline_core ()
#8 0x08069557 in ap_read_request ()
#9 0x08081768 in ap_register_input_filter ()
#10 0x0807b507 in ap_run_process_connection ()
#11 0x0808914b in ap_graceful_stop_signalled ()
#12 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#13 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
#14 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
(gdb) btt 8
[Switching to thread 8 (process 2129)]#0 0xb7c32ef9 in poll () from
/lib/tls/libc.so.6
#0 0xb7c32ef9 in poll () from /lib/tls/libc.so.6
#1 0xb7d10145 in apr_wait_for_io_or_timeout () from /usr/lib/libapr-1.so.0
#2 0xb7d0b247 in apr_socket_sendfile () from /usr/lib/libapr-1.so.0
#3 0x080734d7 in ap_core_output_filter ()
#4 0x08085418 in ap_http_header_filter ()
#5 0x08069d24 in ap_content_length_filter ()
#6 0x08086855 in ap_byterange_filter ()
#7 0xb781b120 in ?? () from /usr/lib/apache2/modules/mod_cache.so
#8 0xb1309398 in ?? ()
#9 0xb1312e08 in ?? ()
#10 0x00000007 in ?? ()
#11 0x00000000 in ?? ()
(gdb) btt 9
[Switching to thread 9 (process 2125)]#0 0xb7b99d51 in kill () from
/lib/tls/libc.so.6
#0 0xb7b99d51 in kill () from /lib/tls/libc.so.6
#1 0x0807c9bb in ap_fatal_signal_child_setup ()
#2 <signal handler called>
#3 0xb77eb15a in modperl_mgv_as_string (my_perl=0x85bc758,
symbol=0x8178190, p=0x88ad7f8, package=0) at modperl_mgv.c:399
#4 0xb77df146 in modperl_callback (my_perl=0x85bc758,
handler=0x88af2a8, p=0x88ad7f8, r=0x88ad830, s=0x80a88c8, args=0x86eaf0c)
at modperl_callback.c:85
#5 0xb77dfb79 in modperl_callback_run_handlers (idx=5, type=4,
r=0x88ad830, c=0x0, s=0x80a88c8, pconf=0x0, plog=0x0, ptemp=0x0,
run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:263
#6 0xb77e00aa in modperl_callback_per_dir (idx=5, r=0x88ad830,
run_mode=MP_HOOK_RUN_ALL) at modperl_callback.c:370
#7 0xb77feb63 in modperl_fixup_handler (r=0x88ad830) at modperl_hooks.c:57
#8 0x0806ff29 in ap_run_fixups ()
#9 0x08084528 in ap_process_request ()
#10 0x080817de in ap_register_input_filter ()
#11 0x0807b507 in ap_run_process_connection ()
#12 0x0808914b in ap_graceful_stop_signalled ()
#13 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#14 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
#15 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
(gdb) btt 10
[Switching to thread 10 (process 2124)]#0 0xb7c32ef9 in poll () from
/lib/tls/libc.so.6
#0 0xb7c32ef9 in poll () from /lib/tls/libc.so.6
#1 0xb7d10145 in apr_wait_for_io_or_timeout () from /usr/lib/libapr-1.so.0
#2 0xb7d0b6b3 in apr_socket_recv () from /usr/lib/libapr-1.so.0
#3 0x0807b756 in ap_lingering_close ()
#4 0x08089153 in ap_graceful_stop_signalled ()
#5 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#6 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
#7 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
(gdb) btt 11
[Switching to thread 11 (process 2123)]#0 0xb7c32ef9 in poll () from
/lib/tls/libc.so.6
#0 0xb7c32ef9 in poll () from /lib/tls/libc.so.6
#1 0xb7d10145 in apr_wait_for_io_or_timeout () from /usr/lib/libapr-1.so.0
#2 0xb7d0b058 in apr_socket_sendv () from /usr/lib/libapr-1.so.0
#3 0x08073145 in ap_bucket_eoc_create ()
#4 0x080739d7 in ap_core_output_filter ()
#5 0x08069d24 in ap_content_length_filter ()
#6 0xb781b4fe in ?? () from /usr/lib/apache2/modules/mod_cache.so
#7 0x0890f298 in ?? ()
#8 0x08905298 in ?? ()
#9 0x08905298 in ?? ()
#10 0x08905298 in ?? ()
#11 0x00000000 in ?? ()
(gdb) btt 12
[Switching to thread 12 (process 2122)]#0 0xb7c32ef9 in poll () from
/lib/tls/libc.so.6
#0 0xb7c32ef9 in poll () from /lib/tls/libc.so.6
#1 0xb7d10145 in apr_wait_for_io_or_timeout () from /usr/lib/libapr-1.so.0
#2 0xb7d0b6b3 in apr_socket_recv () from /usr/lib/libapr-1.so.0
#3 0x0807b756 in ap_lingering_close ()
#4 0x08089153 in ap_graceful_stop_signalled ()
#5 0xb7d10316 in apr_proc_detach () from /usr/lib/libapr-1.so.0
#6 0xb7cad0bd in start_thread () from /lib/tls/libpthread.so.0
#7 0xb7c3c9ee in clone () from /lib/tls/libc.so.6
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Perrin Harkins <pe...@elem.com>.
On 6/4/07, Jani M. <mo...@iki.fi> wrote:
> You are correct in that performance (cpu usage) might be worse with the
> threaded workers. However, scalability is definately much better.
No, it's just the opposite. Using prefork won't save CPU, but it will
save memory, meaning you can run more perl interpreters.
> With threaded processes, I can easily have the same machine run 5x the
> number of threads, with some 400-500 perl interpreters available. This
> is very helpful when you need to be able to serve a large amount of
> concurrent slow(ish) clients.
The normal way to do this is to have your mod_perl server separate
from your static file server, usually by doing a reverse proxy to the
mod_perl server. Then the static server handles doling out bytes to
slow clients. A few variations of this setup are discussed in the
docs.
> Are you sure that this would help in this case? From what I can see, the
> cause of the segfaults appears to be memory corruption, which is likely
> to happen earlier than the actual segfault. If this is the case, would
> the backtrace likely not show any useful information?
Honestly, the person who has done the most work on debugging thread
crashes is Torsten. His advice on how to debug it will be better than
mine. It does seem like people usually solve them by using backtrace
analysis though.
> The reason I ask this is because setting up a full-blown debugging
> environment could be a bit tricky. I might be able to compile a
> debug-enabled version of mod_perl, but doing the same for Apache and/or
> Perl could be a problem. Or are there debug-enabled versions available
> from Debian somewhere?
I don't use Debian, so I couldn't tell you that. Compiling perl,
apache, and mod_perl is not very difficult on a Linux system though.
It takes a little while to compile, but there's no trick to it.
- Perrin
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by "Jani M." <mo...@iki.fi>.
Torsten Foertsch wrote:
> These figures match my observations. It also means that you are not using big
> content generators in the response phase like Catalyst.
Correct. Response content is generated by backend servers, which is then
delivered to clients either from the local cache, or via mod_proxy.
> Did you know this little trick?
>
> In PostReadRequest or so:
> $r->pnotes->{xp}=$r->pool->new;
> $r->push_handlers(PerlFixupHandler=>sub {delete $_[0]->pnotes->{xp}; 0});
No, I did not. Thanks. I'm not sure if I want to (or even need to) use
it though, but it's definitely good to have as an option. Even if it is
hackish :)
> Did you know that you can even share data between Perl interpreters by
> using "threads" and "threads::shared"?
I'm assuming that this will only work for interpreters inside the same
child process? If so, then I don't see that much use for it. If on the
other hand it would work globally across all interpreters, I might have
some use for it.
> already in the parent process before it forks. I am thinking of a patch to
> implement this. But it would mean to abandon growing and shrinking of the
> interpreter pool and the limit of requests per interpreter. But in the end I
> think it would be more reasonable than it is now.
This would be perfect in my use case. I don't need to grow or shrink the
interpreter pool, and limiting the number of requests shouldn't be
needed either - the limit can be set on the apache process side. If you
come up with a patch for this, I'd be happy to give it a try!
Cheers,
-- Jani
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Torsten Foertsch <to...@gmx.net>.
On Monday 04 June 2007 18:43, Jani M. wrote:
> For comparison, a server with one gig of memory can run roughly 600
> preforked workers with some spare memory left for disk caching. Any more
> than this, and performance drops as disk access increases.
These figures match my observations. It also means that you are not using big
content generators in the response phase like Catalyst.
> With threaded processes, I can easily have the same machine run 5x the
> number of threads, with some 400-500 perl interpreters available. This
> is very helpful when you need to be able to serve a large amount of
> concurrent slow(ish) clients.
>
> Obviously the configuration I have uses 'PerlInterpScope handler' to
> maximize the benefit from the interpreters-to-threads ratio.
This is by now the only usable configuration with the worker MPM and it's
really useful. Especially if you allow keepalive requests which is a dont
with prefork MPM. It may have benefits even with "PerlInterpScope request".
Did you know this little trick?
In PostReadRequest or so:
$r->pnotes->{xp}=$r->pool->new;
$r->push_handlers(PerlFixupHandler=>sub {delete $_[0]->pnotes->{xp}; 0});
It binds the interpreter to your request for the whole time from
PostReadRequest to Fixup and hence enables $r->pnotes. It may be useful when
passing complex Perl structures or anonymous coderefs. But be watch out not
to call $r->discard_request_body or so in one of these phases.
But be aware this is very hackish land.
Did you know that you can even share data between Perl interpreters by
using "threads" and "threads::shared"?
The main problem with the memory consumption of perl interpreters with worker
MPM is that they are not shared by copy-on-write. There is one main
interpeter per process that is cloned in ChildInit (or later). I think
copy-on-write can be used for multiple interpreters if they are cloned
already in the parent process before it forks. I am thinking of a patch to
implement this. But it would mean to abandon growing and shrinking of the
interpreter pool and the limit of requests per interpreter. But in the end I
think it would be more reasonable than it is now.
Does this make sense?
Torsten
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by "Jani M." <mo...@iki.fi>.
Perrin Harkins wrote:
> Are you sure about that? On Linux, prefork is likely to perform
> better. Threads will use up a lot more memory, and be slow to spawn.
> I think you should try a little benchmarking before you put a lot more
> time into using threads.
You are correct in that performance (cpu usage) might be worse with the
threaded workers. However, scalability is definately much better.
For comparison, a server with one gig of memory can run roughly 600
preforked workers with some spare memory left for disk caching. Any more
than this, and performance drops as disk access increases.
With threaded processes, I can easily have the same machine run 5x the
number of threads, with some 400-500 perl interpreters available. This
is very helpful when you need to be able to serve a large amount of
concurrent slow(ish) clients.
Obviously the configuration I have uses 'PerlInterpScope handler' to
maximize the benefit from the interpreters-to-threads ratio.
> If it turns out you do need to get threads to work, the next step in
> fixing segfaults is typically to get a backtrace. There are
> instructions for doing this in the mod_perl docs.
Are you sure that this would help in this case? From what I can see, the
cause of the segfaults appears to be memory corruption, which is likely
to happen earlier than the actual segfault. If this is the case, would
the backtrace likely not show any useful information?
The reason I ask this is because setting up a full-blown debugging
environment could be a bit tricky. I might be able to compile a
debug-enabled version of mod_perl, but doing the same for Apache and/or
Perl could be a problem. Or are there debug-enabled versions available
from Debian somewhere?
I have been creating a stripped-down-model of this setup in a test
environment, in an attempt to be able to pin point what exactly causes
the crashes. Unfortunately I haven't yet been able to create a setup
which would duplicate the segfaults, despite that fact that the test
environment already implements nearly 100% of the functionality that the
production system has. I'll keep on trying though.
Thanks,
-- Jani
Re: [mp2] Segmentation faults with threaded worker-mpm
Posted by Perrin Harkins <pe...@elem.com>.
On 6/3/07, Jani M. <mo...@iki.fi> wrote:
> The problems start only when two or more interpreters per
> process is running - unfortunately, this is exactly what I would need
> for scalability and performance.
Are you sure about that? On Linux, prefork is likely to perform
better. Threads will use up a lot more memory, and be slow to spawn.
I think you should try a little benchmarking before you put a lot more
time into using threads.
If it turns out you do need to get threads to work, the next step in
fixing segfaults is typically to get a backtrace. There are
instructions for doing this in the mod_perl docs.
- Perrin