You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2017/03/15 12:07:37 UTC
[Bug 60869] New: Workers locking up instead finishing during eg.
graceful reload
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
Bug ID: 60869
Summary: Workers locking up instead finishing during eg.
graceful reload
Product: Apache httpd-2
Version: 2.4.25
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_http2
Assignee: bugs@httpd.apache.org
Reporter: bug@bszx.eu
Target Milestone: ---
Created attachment 34829
--> https://bz.apache.org/bugzilla/attachment.cgi?id=34829&action=edit
server status
Tested on prefork mpm and event mpm.
How to reproduce:
- enable mod_http2
- start apache
$ ./apachectl start
- reload apache
$ ./apachectl graceful
After this all existing workers are locked up in finishing state like in
attached server status:
Srv PID Acc M CPU SS Req Conn Child Slot
Client Protocol VHost Request
0-0 9684 0/0/0 G 0.00 52 0 0.0 0.00 0.00
1-0 9685 0/0/0 G 0.00 52 0 0.0 0.00 0.00
2-0 9686 0/3/3 G 0.00 26 0 0.0 0.00 0.00
127.0.0.1 http/1.1 127.0.0.1:8081 GET /favicon.ico HTTP/1.1
3-0 9687 0/0/0 G 0.00 52 0 0.0 0.00 0.00
4-0 9688 0/0/0 G 0.00 52 0 0.0 0.00 0.00
5-0 9695 0/0/0 G 0.00 31 0 0.0 0.00 0.00
6-1 9701 0/0/0 W 0.00 0 0 0.0 0.00 0.00
127.0.0.1 http/1.1 127.0.0.1:8081 GET /server-status HTTP/1.1
$ strace -f -p 9684
strace: Process 9684 attached with 2 threads
[pid 9690] futex(0x1208718, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 9684] futex(0x1208714, FUTEX_WAIT_PRIVATE, 12, NULL
$ gdb -p 9684
(gdb) thread apply all bt
Thread 2 (Thread 0x7f05a9c93700 (LWP 9690)):
#0 0x00007f05ae69f9f6 in futex_wait_cancelable (private=<optimized out>,
expected=0, futex_word=0x1208718)
at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x12086a0, cond=0x12086f0)
at pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x12086f0, mutex=0x12086a0) at
pthread_cond_wait.c:655
#3 0x00007f05ab155ba1 in get_mplx_next (worker=0x1208790, ctx=0x12085a0,
ptask=0x7f05a9c92ea8, psticky=0x7f05a9c92eb4) at h2_workers.c:160
#4 0x00007f05ab155049 in execute (thread=0x12087d0, wctx=0x1208790) at
h2_worker.c:43
#5 0x00007f05ae699550 in start_thread (arg=0x7f05a9c93700) at
pthread_create.c:456
#6 0x00007f05ae1d49af in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:94
#7 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7f05af7af540 (LWP 9684)):
#0 0x00007f05ae69f539 in futex_wait (private=<optimized out>, expected=12,
futex_word=0x1208714)
at ../sysdeps/unix/sysv/linux/futex-internal.h:61
#1 futex_wait_simple (private=<optimized out>, expected=12,
futex_word=0x1208714) at ../sysdeps/nptl/futex-internal.h:135
#2 __pthread_cond_destroy (cond=0x12086f0) at pthread_cond_destroy.c:54
#3 0x00007f05aead226e in run_cleanups (cref=<optimized out>) at
memory/unix/apr_pools.c:2352
#4 apr_pool_destroy (pool=0x1208528) at memory/unix/apr_pools.c:814
#5 0x00007f05aead2245 in apr_pool_destroy (pool=0x1204508) at
memory/unix/apr_pools.c:811
#6 0x00007f05aace3c89 in clean_child_exit (code=0) at prefork.c:227
#7 0x00007f05aace5052 in child_main (child_num_arg=0, child_bucket=0) at
prefork.c:744
#8 0x00007f05aace524e in make_child (s=0x118e3d0, slot=0, bucket=0) at
prefork.c:824
#9 0x00007f05aace52ed in startup_children (number_to_start=5) at prefork.c:843
#10 0x00007f05aace58bf in prefork_run (_pconf=0x1167138, plog=0x1194358,
s=0x118e3d0) at prefork.c:1010
#11 0x0000000000434033 in ap_run_mpm (pconf=0x1167138, plog=0x1194358,
s=0x118e3d0) at mpm_common.c:94
#12 0x000000000042b1b3 in main (argc=3, argv=0x7ffc5e108828) at main.c:783
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
--- Comment #2 from Bartek Szady <bu...@bszx.eu> ---
Created attachment 34830
--> https://bz.apache.org/bugzilla/attachment.cgi?id=34830&action=edit
server status with prefork mpm
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
--- Comment #3 from Bartek Szady <bu...@bszx.eu> ---
Created attachment 34831
--> https://bz.apache.org/bugzilla/attachment.cgi?id=34831&action=edit
server status with event mpm
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
--- Comment #5 from Bartek Szady <bu...@bszx.eu> ---
I've copied the Apache 2.4.x subversion branch rev 1787046 (and srclib/apr
branch 1.5.x rev 1787048; srclib/apr-util branch 1.5.x rev 1787048) tested the
result and the issue is still there.
Test results with prefork_mpm (server status is attached):
$ strace -f -p 23435
strace: Process 23435 attached with 2 threads
[pid 23437] futex(0x77f718, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23435] futex(0x77f714, FUTEX_WAIT_PRIVATE, 12, NULL
$ gdb -p 23435
(gdb) thread apply all bt
Thread 2 (Thread 0x7fc000a5c700 (LWP 23437)):
#0 0x00007fc0054689f6 in futex_wait_cancelable (private=<optimized out>,
expected=0, futex_word=0x77f718)
at ../sysdeps/unix/sysv/linux/futex-internal.h:88
#1 __pthread_cond_wait_common (abstime=0x0, mutex=0x77f6a0, cond=0x77f6f0) at
pthread_cond_wait.c:502
#2 __pthread_cond_wait (cond=0x77f6f0, mutex=0x77f6a0) at
pthread_cond_wait.c:655
#3 0x00007fc001f1eba1 in get_mplx_next (worker=0x77f790, ctx=0x77f5a0,
ptask=0x7fc000a5bea8, psticky=0x7fc000a5beb4) at h2_workers.c:160
#4 0x00007fc001f1e049 in execute (thread=0x77f7d0, wctx=0x77f790) at
h2_worker.c:43
#5 0x00007fc005462550 in start_thread (arg=0x7fc000a5c700) at
pthread_create.c:456
#6 0x00007fc004f9d9af in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:94
#7 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fc006578540 (LWP 23435)):
#0 0x00007fc005468539 in futex_wait (private=<optimized out>, expected=12,
futex_word=0x77f714)
at ../sysdeps/unix/sysv/linux/futex-internal.h:61
#1 futex_wait_simple (private=<optimized out>, expected=12,
futex_word=0x77f714) at ../sysdeps/nptl/futex-internal.h:135
#2 __pthread_cond_destroy (cond=0x77f6f0) at pthread_cond_destroy.c:54
#3 0x00007fc00589b26e in run_cleanups (cref=<optimized out>) at
memory/unix/apr_pools.c:2352
#4 apr_pool_destroy (pool=0x77f528) at memory/unix/apr_pools.c:814
#5 0x00007fc00589b245 in apr_pool_destroy (pool=0x77b508) at
memory/unix/apr_pools.c:811
#6 0x00007fc001aacc89 in clean_child_exit (code=0) at prefork.c:227
#7 0x00007fc001aae052 in child_main (child_num_arg=0, child_bucket=0) at
prefork.c:744
#8 0x00007fc001aae24e in make_child (s=0x7053d0, slot=0, bucket=0) at
prefork.c:824
#9 0x00007fc001aae2ed in startup_children (number_to_start=5) at prefork.c:843
#10 0x00007fc001aae8bf in prefork_run (_pconf=0x6de138, plog=0x70b358,
s=0x7053d0) at prefork.c:1010
#11 0x0000000000434033 in ap_run_mpm ()
#12 0x000000000042b1b3 in main ()
Test results with event_mpm (server status and backtrace is attached):
$ strace -f -p 23507
strace: Process 23507 attached with 26 threads
[pid 23551] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23555] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23548] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23545] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23544] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23541] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23538] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23535] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23533] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23530] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23527] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23525] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23524] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23522] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23521] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23519] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23518] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23517] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23516] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23515] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23514] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23513] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23512] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23510] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23509] futex(0x1c29ee8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 23507] futex(0x1c29ee4, FUTEX_WAIT_PRIVATE, 204, NULL
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
--- Comment #9 from Stefan Eissing <st...@eissing.org> ---
Well, no harm done and thanks for re-verifying. I will the commit the attached
patch, so it will be part of the next Apache, as well as the next mod_http2
github release.
Seems that on most system, conditionals wake up threads on destruction, but not
on yours. But I am not sure if it is guarantueed that way. The patch makes the
behaviour defined which is for the better.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
--- Comment #4 from Bartek Szady <bu...@bszx.eu> ---
Created attachment 34832
--> https://bz.apache.org/bugzilla/attachment.cgi?id=34832&action=edit
backtrace with event mpm
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
--- Comment #6 from Stefan Eissing <st...@eissing.org> ---
Created attachment 34833
--> https://bz.apache.org/bugzilla/attachment.cgi?id=34833&action=edit
test patch for worker exit
Can you check if the attached patch helps in your scenario?
graceful restarts are part of the normal h2 test suite, so on my systems the
current implementation works fine. Something is different on your machine. You
are not in any way affected by something like:
https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64 maybe?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
--- Comment #1 from Stefan Eissing <st...@eissing.org> ---
Bartek, thanks for you bug report. There have been fixes done in the meantime
regarding mod_http2 and graceful restart. Are you able to test your setup with
a copy of the Apache 2.4.x subversion branch or are you able to build and test
the module from https://github.com/icing/mod_h2
That would be helpful.
Unrelated to this: have you tried using another mpm module than prefork?
mpm_event is highly recommended when you want to use HTTP/2 since prefork
cannot really benefit from it.
-Stefan
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
Stefan Eissing <st...@eissing.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #10 from Stefan Eissing <st...@eissing.org> ---
Fixed in r1787141.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
--- Comment #8 from Bartek Szady <bu...@bszx.eu> ---
Another mistake :-(
I've retested it once again carefully (I've ensured that mod_http2 was enabled)
and noticed that:
Rev 1787046 with attached patch works properly.
Rev 1787046 without attached patch locks up.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 60869] Workers locking up instead finishing during eg. graceful
reload
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=60869
--- Comment #7 from Bartek Szady <bu...@bszx.eu> ---
I've made a mistake during the second test (old mod_http2 was loaded).
Rev 1787046 works properly with and without attached patch.
Sorry for the noise.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org