You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jan Kaluža <jk...@redhat.com> on 2013/05/07 08:37:06 UTC
prefork mpm crashes on SIGINT [possible patch?]
Hi,
to reproduce it, I start httpd-2.4.4 like this:
/usr/sbin/httpd -k start -DFOREGROUND
and in another terminal I run ab like this:
ab -n 150000 -c 4 http://127.0.0.1/index.php
Index.php is just dummy script with phpinfo.
When "ab" is benchmarking, I kill httpd using "ctrl+c". In 4/10 tries it
crashes with following backtrace [1].
If gdb does not lie, there is race condition in signal handling. I'm not
sure if anything changed in this code between 2.2 and 2.4, but it looks
like 2.2 was handling this situation somehow better. I think this
situation can lead to various problems like deadlocks.
I've been able to workaround this using the attached patch, but I don't
think this is the best way how to fix this problem. It would be better
to find out why there are more signals called in a row as a result of
single SIGINT signal sent to main process.
I will try to do that myself, but I was thinking there could be people
more familiar with this part of code and with httpd's children lifecycle
reading this list.
[1]
#0 0x00007fe74b7d2ba5 in raise () from /lib64/libc.so.6
#1 0x00007fe74b7d4358 in abort () from /lib64/libc.so.6
#2 0x00007fe74b81259b in __libc_message () from /lib64/libc.so.6
#3 0x00007fe74b819a8e in _int_free () from /lib64/libc.so.6
#4 0x00007fe74bf8e85c in allocator_free (node=0x7fe74bb4e798
<main_arena+88>, allocator=0x7fe74e7ab670) at memory/unix/apr_pools.c:430
#5 apr_pool_destroy (pool=0x7fe74e7ab768) at memory/unix/apr_pools.c:856
#6 0x00007fe74308c1be in clean_child_exit (code=code@entry=0) at
prefork.c:218
#7 0x00007fe74308c9cb in just_die (sig=<optimized out>) at prefork.c:344
#8 <signal handler called>
#9 0x00007fe74d06d541 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
#10 0x00007fe74b7d5df1 in __run_exit_handlers () from /lib64/libc.so.6
#11 0x00007fe74b7d5e75 in exit () from /lib64/libc.so.6
#12 0x00007fe74308c1db in clean_child_exit (code=code@entry=0) at
prefork.c:227
#13 0x00007fe74308c5f5 in child_main
(child_num_arg=child_num_arg@entry=2) at prefork.c:725
#14 0x00007fe74308c94c in make_child (s=0x7fe74e5a5368,
slot=slot@entry=2) at prefork.c:800
#15 0x00007fe74308c9a6 in startup_children (number_to_start=3) at
prefork.c:818
#16 0x00007fe74308d856 in prefork_run (_pconf=<optimized out>,
plog=0x7fe74e5d74f8, s=0x7fe74e5a5368) at prefork.c:976
#17 0x00007fe74d2a5d4e in ap_run_mpm ()
#18 0x00007fe74d29f3fa in main ()
Thanks for help,
Jan Kaluza
Re: prefork mpm crashes on SIGINT [possible patch?]
Posted by Jan Kaluža <jk...@redhat.com>.
On 08/14/2014 04:22 PM, Jan Kaluža wrote:
> Hi,
>
> I'm resurrecting this old thread, because I hesitate to do changes in
> signal handling without any ack of someone else.
Committed in r1618555.
Jan Kaluza
>
> This bug is more problematic in the context of docker [1] where people
> tend to run "httpd -DFOREGROUND", because in docker, you should run
> applications on foreground and docker will manage the application itself.
>
> Note that from time to time depending on luck, this is reproducible even
> without any load and the server. Just "httpd -DFOREGROUND" and then ctrl+c.
>
> [1] https://www.docker.com/
>
> Is anyone against the patch I sent last year in previous email in this
> thread below?
>
> Regards,
> Jan Kaluza
>
> On 05/09/2013 09:03 AM, Jan Kaluža wrote:
>> I think I have proper solution for the crashes mentioned in previous
>> mail (see the attached patch):
>>
>> 1. Ignore SIGINT and SIGTERM in clean_child_exit. Handlers of these two
>> signals would call clean_child_exit again and there is no reason to
>> handle them once we are exiting.
>>
>> 2. Ignore SIGINT completely in children. I'm not sure I see valid
>> use-case in sending SIGINT to child process, but it's causing crashes
>> when running with -DFOREGROUND. See the comment in attached patch for
>> more info.
>>
>> I was not able to reproduce the crash with this patch.
>>
>> On 05/07/2013 08:37 AM, Jan Kaluža wrote:
>>> Hi,
>>>
>>> to reproduce it, I start httpd-2.4.4 like this:
>>>
>>> /usr/sbin/httpd -k start -DFOREGROUND
>>>
>>> and in another terminal I run ab like this:
>>>
>>> ab -n 150000 -c 4 http://127.0.0.1/index.php
>>>
>>> Index.php is just dummy script with phpinfo.
>>>
>>> When "ab" is benchmarking, I kill httpd using "ctrl+c". In 4/10 tries it
>>> crashes with following backtrace [1].
>>> If gdb does not lie, there is race condition in signal handling. I'm not
>>> sure if anything changed in this code between 2.2 and 2.4, but it looks
>>> like 2.2 was handling this situation somehow better. I think this
>>> situation can lead to various problems like deadlocks.
>>>
>>> I've been able to workaround this using the attached patch, but I don't
>>> think this is the best way how to fix this problem. It would be better
>>> to find out why there are more signals called in a row as a result of
>>> single SIGINT signal sent to main process.
>>>
>>> I will try to do that myself, but I was thinking there could be people
>>> more familiar with this part of code and with httpd's children lifecycle
>>> reading this list.
>>>
>>> [1]
>>>
>>> #0 0x00007fe74b7d2ba5 in raise () from /lib64/libc.so.6
>>> #1 0x00007fe74b7d4358 in abort () from /lib64/libc.so.6
>>> #2 0x00007fe74b81259b in __libc_message () from /lib64/libc.so.6
>>> #3 0x00007fe74b819a8e in _int_free () from /lib64/libc.so.6
>>> #4 0x00007fe74bf8e85c in allocator_free (node=0x7fe74bb4e798
>>> <main_arena+88>, allocator=0x7fe74e7ab670) at
>>> memory/unix/apr_pools.c:430
>>> #5 apr_pool_destroy (pool=0x7fe74e7ab768) at
>>> memory/unix/apr_pools.c:856
>>> #6 0x00007fe74308c1be in clean_child_exit (code=code@entry=0) at
>>> prefork.c:218
>>> #7 0x00007fe74308c9cb in just_die (sig=<optimized out>) at
>>> prefork.c:344
>>> #8 <signal handler called>
>>> #9 0x00007fe74d06d541 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
>>> #10 0x00007fe74b7d5df1 in __run_exit_handlers () from /lib64/libc.so.6
>>> #11 0x00007fe74b7d5e75 in exit () from /lib64/libc.so.6
>>> #12 0x00007fe74308c1db in clean_child_exit (code=code@entry=0) at
>>> prefork.c:227
>>> #13 0x00007fe74308c5f5 in child_main
>>> (child_num_arg=child_num_arg@entry=2) at prefork.c:725
>>> #14 0x00007fe74308c94c in make_child (s=0x7fe74e5a5368,
>>> slot=slot@entry=2) at prefork.c:800
>>> #15 0x00007fe74308c9a6 in startup_children (number_to_start=3) at
>>> prefork.c:818
>>> #16 0x00007fe74308d856 in prefork_run (_pconf=<optimized out>,
>>> plog=0x7fe74e5d74f8, s=0x7fe74e5a5368) at prefork.c:976
>>> #17 0x00007fe74d2a5d4e in ap_run_mpm ()
>>> #18 0x00007fe74d29f3fa in main ()
>>>
>>> Thanks for help,
>>> Jan Kaluza
>>
>> Regards,
>> Jan Kaluza
>
Re: prefork mpm crashes on SIGINT [possible patch?]
Posted by Jan Kaluža <jk...@redhat.com>.
Hi,
I'm resurrecting this old thread, because I hesitate to do changes in
signal handling without any ack of someone else.
This bug is more problematic in the context of docker [1] where people
tend to run "httpd -DFOREGROUND", because in docker, you should run
applications on foreground and docker will manage the application itself.
Note that from time to time depending on luck, this is reproducible even
without any load and the server. Just "httpd -DFOREGROUND" and then ctrl+c.
[1] https://www.docker.com/
Is anyone against the patch I sent last year in previous email in this
thread below?
Regards,
Jan Kaluza
On 05/09/2013 09:03 AM, Jan Kaluža wrote:
> I think I have proper solution for the crashes mentioned in previous
> mail (see the attached patch):
>
> 1. Ignore SIGINT and SIGTERM in clean_child_exit. Handlers of these two
> signals would call clean_child_exit again and there is no reason to
> handle them once we are exiting.
>
> 2. Ignore SIGINT completely in children. I'm not sure I see valid
> use-case in sending SIGINT to child process, but it's causing crashes
> when running with -DFOREGROUND. See the comment in attached patch for
> more info.
>
> I was not able to reproduce the crash with this patch.
>
> On 05/07/2013 08:37 AM, Jan Kaluža wrote:
>> Hi,
>>
>> to reproduce it, I start httpd-2.4.4 like this:
>>
>> /usr/sbin/httpd -k start -DFOREGROUND
>>
>> and in another terminal I run ab like this:
>>
>> ab -n 150000 -c 4 http://127.0.0.1/index.php
>>
>> Index.php is just dummy script with phpinfo.
>>
>> When "ab" is benchmarking, I kill httpd using "ctrl+c". In 4/10 tries it
>> crashes with following backtrace [1].
>> If gdb does not lie, there is race condition in signal handling. I'm not
>> sure if anything changed in this code between 2.2 and 2.4, but it looks
>> like 2.2 was handling this situation somehow better. I think this
>> situation can lead to various problems like deadlocks.
>>
>> I've been able to workaround this using the attached patch, but I don't
>> think this is the best way how to fix this problem. It would be better
>> to find out why there are more signals called in a row as a result of
>> single SIGINT signal sent to main process.
>>
>> I will try to do that myself, but I was thinking there could be people
>> more familiar with this part of code and with httpd's children lifecycle
>> reading this list.
>>
>> [1]
>>
>> #0 0x00007fe74b7d2ba5 in raise () from /lib64/libc.so.6
>> #1 0x00007fe74b7d4358 in abort () from /lib64/libc.so.6
>> #2 0x00007fe74b81259b in __libc_message () from /lib64/libc.so.6
>> #3 0x00007fe74b819a8e in _int_free () from /lib64/libc.so.6
>> #4 0x00007fe74bf8e85c in allocator_free (node=0x7fe74bb4e798
>> <main_arena+88>, allocator=0x7fe74e7ab670) at memory/unix/apr_pools.c:430
>> #5 apr_pool_destroy (pool=0x7fe74e7ab768) at memory/unix/apr_pools.c:856
>> #6 0x00007fe74308c1be in clean_child_exit (code=code@entry=0) at
>> prefork.c:218
>> #7 0x00007fe74308c9cb in just_die (sig=<optimized out>) at prefork.c:344
>> #8 <signal handler called>
>> #9 0x00007fe74d06d541 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
>> #10 0x00007fe74b7d5df1 in __run_exit_handlers () from /lib64/libc.so.6
>> #11 0x00007fe74b7d5e75 in exit () from /lib64/libc.so.6
>> #12 0x00007fe74308c1db in clean_child_exit (code=code@entry=0) at
>> prefork.c:227
>> #13 0x00007fe74308c5f5 in child_main
>> (child_num_arg=child_num_arg@entry=2) at prefork.c:725
>> #14 0x00007fe74308c94c in make_child (s=0x7fe74e5a5368,
>> slot=slot@entry=2) at prefork.c:800
>> #15 0x00007fe74308c9a6 in startup_children (number_to_start=3) at
>> prefork.c:818
>> #16 0x00007fe74308d856 in prefork_run (_pconf=<optimized out>,
>> plog=0x7fe74e5d74f8, s=0x7fe74e5a5368) at prefork.c:976
>> #17 0x00007fe74d2a5d4e in ap_run_mpm ()
>> #18 0x00007fe74d29f3fa in main ()
>>
>> Thanks for help,
>> Jan Kaluza
>
> Regards,
> Jan Kaluza
Re: prefork mpm crashes on SIGINT [possible patch?]
Posted by Jan Kaluža <jk...@redhat.com>.
I think I have proper solution for the crashes mentioned in previous
mail (see the attached patch):
1. Ignore SIGINT and SIGTERM in clean_child_exit. Handlers of these two
signals would call clean_child_exit again and there is no reason to
handle them once we are exiting.
2. Ignore SIGINT completely in children. I'm not sure I see valid
use-case in sending SIGINT to child process, but it's causing crashes
when running with -DFOREGROUND. See the comment in attached patch for
more info.
I was not able to reproduce the crash with this patch.
On 05/07/2013 08:37 AM, Jan Kaluža wrote:
> Hi,
>
> to reproduce it, I start httpd-2.4.4 like this:
>
> /usr/sbin/httpd -k start -DFOREGROUND
>
> and in another terminal I run ab like this:
>
> ab -n 150000 -c 4 http://127.0.0.1/index.php
>
> Index.php is just dummy script with phpinfo.
>
> When "ab" is benchmarking, I kill httpd using "ctrl+c". In 4/10 tries it
> crashes with following backtrace [1].
> If gdb does not lie, there is race condition in signal handling. I'm not
> sure if anything changed in this code between 2.2 and 2.4, but it looks
> like 2.2 was handling this situation somehow better. I think this
> situation can lead to various problems like deadlocks.
>
> I've been able to workaround this using the attached patch, but I don't
> think this is the best way how to fix this problem. It would be better
> to find out why there are more signals called in a row as a result of
> single SIGINT signal sent to main process.
>
> I will try to do that myself, but I was thinking there could be people
> more familiar with this part of code and with httpd's children lifecycle
> reading this list.
>
> [1]
>
> #0 0x00007fe74b7d2ba5 in raise () from /lib64/libc.so.6
> #1 0x00007fe74b7d4358 in abort () from /lib64/libc.so.6
> #2 0x00007fe74b81259b in __libc_message () from /lib64/libc.so.6
> #3 0x00007fe74b819a8e in _int_free () from /lib64/libc.so.6
> #4 0x00007fe74bf8e85c in allocator_free (node=0x7fe74bb4e798
> <main_arena+88>, allocator=0x7fe74e7ab670) at memory/unix/apr_pools.c:430
> #5 apr_pool_destroy (pool=0x7fe74e7ab768) at memory/unix/apr_pools.c:856
> #6 0x00007fe74308c1be in clean_child_exit (code=code@entry=0) at
> prefork.c:218
> #7 0x00007fe74308c9cb in just_die (sig=<optimized out>) at prefork.c:344
> #8 <signal handler called>
> #9 0x00007fe74d06d541 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
> #10 0x00007fe74b7d5df1 in __run_exit_handlers () from /lib64/libc.so.6
> #11 0x00007fe74b7d5e75 in exit () from /lib64/libc.so.6
> #12 0x00007fe74308c1db in clean_child_exit (code=code@entry=0) at
> prefork.c:227
> #13 0x00007fe74308c5f5 in child_main
> (child_num_arg=child_num_arg@entry=2) at prefork.c:725
> #14 0x00007fe74308c94c in make_child (s=0x7fe74e5a5368,
> slot=slot@entry=2) at prefork.c:800
> #15 0x00007fe74308c9a6 in startup_children (number_to_start=3) at
> prefork.c:818
> #16 0x00007fe74308d856 in prefork_run (_pconf=<optimized out>,
> plog=0x7fe74e5d74f8, s=0x7fe74e5a5368) at prefork.c:976
> #17 0x00007fe74d2a5d4e in ap_run_mpm ()
> #18 0x00007fe74d29f3fa in main ()
>
> Thanks for help,
> Jan Kaluza
Regards,
Jan Kaluza