You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2006/02/27 17:31:06 UTC
DO NOT REPLY [Bug 38793] New: - Segfault when backend failes, threading issue
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
Summary: Segfault when backend failes, threading issue
Product: Apache httpd-2
Version: 2.2.0
Platform: PC
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_proxy
AssignedTo: bugs@httpd.apache.org
ReportedBy: mk-asf@gigacodes.de
We run apache22 as reverse proxy to a backend application server (apache1.3).
When we put apache22 under load ( ab -k -c 200 ) and restart the backend server,
there's a good chance of a segmentation fault.
We've done multiple runs and all Segfault backtraces shared the
proxy_http_handler in mod_proxy_http.c with calls between the comments "Step
One" and "Step Six". The attached backtrace is the most common one.
My guess is, that when the first connection fails, it clears pooled
connections/ressources currently being used by other threads that are already in
the code between "Step One" and "Step Six", so their backend->connection gets
nulled in the middle of doing something. In my opinion backend->connection
should never be 0 in this part of the code, since all failed functions would
goto cleanup. However backtrace shows them 0 as well as some cond breakpoints.
(Although sometimes breakpoints get passed, but thread segfaults anyway).
#0 0x08095f79 in ap_proxy_make_fake_req (c=0x0, r=0x19520540) at proxy_util.c:351
rp = (request_rec *) 0x94d53f0
#1 0x0809b6ca in ap_proxy_http_process_response (p=0x94d42f8, r=0x19520540,
backend=0x917c0e0, origin=0x0, conf=0x91069d8,
server_portstr=0xaef82190 "") at mod_proxy_http.c:1206
c = (conn_rec *) 0x94d44e8
buffer = '\0' <repeats 3420 times>,
"\211\003_\000\211\003_\000H\037\uffff\uffff\uffffG]\000\000\000\000\000\uffff)`\000\023\000\000\000\uffff\016\uffff\uffff\000\000\000\000\000\000\000\000\uffff\036\uffff\uffff\f\000\000\000\024\000\000\000\uffff!\003D\uffff\016\uffff\uffff\uffff\016\uffff\uffff\uffff\036\uffff\uffff\n\000\000\000\003\000\000\001\207F\000\000\023\000\000\000\uffff\016\uffff\uffff\000\020\000\000\f\000\000\000\uffff\036\uffff\uffff\f\000\000\000\uffff\016\uffff\uffff\001",
'\0' <repeats 19 times>,
"\024\000\000\000\003\000\002\000\uffff!\003D\207F\000\000\000\000\000\000\001\000\000\000\024\000\001",
'\0' <repeats 16 times>,
"\001\024\000\006\000\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffffd\r\000\000d\r\000\000@\000\000\000\024\000\002\000\uffff!\003D\207"...
buf = 0xa <Address 0xa out of bounds>
keepchar = 100 'd'
rp = (request_rec *) 0x95037d8
e = (apr_bucket *) 0x312e302e
bb = (apr_bucket_brigade *) 0x94d53f0
len = -1
backasswards = 774909495
interim_response = 842082336
pread_len = 0
save_table = (apr_table_t *) 0x72656b72
#2 0x0809c7e2 in proxy_http_handler (r=0x19520540, worker=0x9106db0,
conf=0x91069d8, url=0x94d4ff0 "/test/output.php?delay=10",
proxyname=0x0, proxyport=0) at mod_proxy_http.c:1712
status = 0
server_portstr = "\000N\r\b]\005\000\000\a\000\000\000\000\000\000\000
e\030\t\uffffDM\t@\005R\031\000\000\000"
scheme = 0x94d4f90 "http"
proxy_function = 0x80d5ff9 "HTTP"
u = 0x195217fa "://127.0.0.1:8081/test/output.php?delay=10"
backend = (proxy_conn_rec *) 0x917c0e0
is_ssl = 0
p = (apr_pool_t *) 0x94d42f8
c = (conn_rec *) 0x94d44e8
uri = (apr_uri_t *) 0x94d4f60
#3 0x080951d5 in proxy_run_scheme_handler (r=0x19520540, worker=0x9106db0,
conf=0x91069d8,
url=0x195217f6 "http://127.0.0.1:8081/test/output.php?delay=10",
proxyhost=0x0, proxyport=0) at mod_proxy.c:1936
pHook = (proxy_LINK_scheme_handler_t *) 0x91dd358
n = 0
rv = 424810486
#4 0x08092a3d in proxy_handler (r=0x19520540) at mod_proxy.c:739
#5 0x0807cdcd in ap_run_handler (r=0x19520540) at config.c:157
#6 0x0807d50d in ap_invoke_handler (r=0x19520540) at config.c:371
#7 0x080b49d2 in ap_process_request (r=0x19520540) at http_request.c:258
#8 0x080b1ee8 in ap_process_http_connection (c=0x94d44e8) at http_core.c:171
#9 0x08084264 in ap_run_process_connection (c=0x94d44e8) at connection.c:43
#10 0x08084693 in ap_process_connection (c=0x94d44e8, csd=0x94d4338) at
connection.c:178
#11 0x080bddcb in process_socket (p=0x94d42f8, sock=0x94d4338, my_child_num=0,
my_thread_num=12, bucket_alloc=0x1951c4f8)
at worker.c:531
#12 0x080be5d4 in worker_thread (thd=0x91a05c0, dummy=0x91dddc8) at worker.c:876
#13 0xb7ef7868 in dummy_worker (opaque=0x91a05c0) at threadproc/unix/thread.c:138
#14 0x006bb341 in start_thread () from /lib/tls/libpthread.so.0
#15 0x005e36fe in clone () from /lib/tls/libc.so.6
as mentioned this is the most common one but we even got segfaults here:
#0 0xb7ed2246 in allocator_free (allocator=0x0, node=0x9b15110) at
memory/unix/apr_pools.c:319
#1 0xb7ed2224 in apr_allocator_free (allocator=0x0, node=0x9b15110) at
memory/unix/apr_pools.c:384
#2 0xb7fbd81f in apr_bucket_free (mem=0x9b15138) at buckets/apr_buckets_alloc.c:182
#3 0xb7fbda6a in socket_bucket_read (a=0x9b011c8, str=0xb6a47f40,
len=0xb6a47f3c, block=APR_BLOCK_READ)
at buckets/apr_buckets_socket.c:71
#4 0xb7fbe933 in apr_brigade_split_line (bbOut=0x9af8180, bbIn=0x9af81a8,
block=APR_BLOCK_READ, maxbytes=8192)
at buckets/apr_brigade.c:292
#5 0x0807b024 in ap_core_input_filter (f=0x9af71d0, b=0x9af8180,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
at core_filters.c:171
#6 0x080878e3 in ap_get_brigade (next=0x9af71d0, bb=0x9af8180,
mode=AP_MODE_GETLINE, block=APR_BLOCK_READ, readbytes=0)
at util_filter.c:489
#7 0x0806e358 in ap_rgetline_core (s=0xb6a480a4, n=8192, read=0xb6a4809c,
r=0x9af7400, fold=0, bb=0x9af8180) at protocol.c:222
#8 0x0806e78f in ap_getline (s=0xb6a48120 "", n=8192, r=0x9af7400, fold=0) at
protocol.c:463
#9 0x0809b70a in ap_proxy_http_process_response (p=0x9b04a98, r=0x9b11150,
backend=0x9a94e80, origin=0x9af6f90, conf=0x9a1f9d8,
server_portstr=0xb6a4a190 "") at mod_proxy_http.c:1214
#10 0x0809c7e2 in proxy_http_handler (r=0x9b11150, worker=0x9a1fdb0,
conf=0x9a1f9d8, url=0x9b05018 "/test/output.php?delay=1",
proxyname=0x0, proxyport=0) at mod_proxy_http.c:1712
#11 0x080951d5 in proxy_run_scheme_handler (r=0x9b11150, worker=0x9a1fdb0,
conf=0x9a1f9d8,
url=0x9b123fe "http://127.0.0.1:8081/test/output.php?delay=1",
proxyhost=0x0, proxyport=0) at mod_proxy.c:1936
#12 0x08092a3d in proxy_handler (r=0x9b11150) at mod_proxy.c:739
#13 0x0807cdcd in ap_run_handler (r=0x9b11150) at config.c:157
#14 0x0807d50d in ap_invoke_handler (r=0x9b11150) at config.c:371
#15 0x080b49d2 in ap_process_request (r=0x9b11150) at http_request.c:258
#16 0x080b1ee8 in ap_process_http_connection (c=0x9b04c88) at http_core.c:171
#17 0x08084264 in ap_run_process_connection (c=0x9b04c88) at connection.c:43
#18 0x08084693 in ap_process_connection (c=0x9b04c88, csd=0x9b04ad8) at
connection.c:178
#19 0x080bddcb in process_socket (p=0x9b04a98, sock=0x9b04ad8, my_child_num=0,
my_thread_num=1, bucket_alloc=0x9b06aa0)
at worker.c:531
#20 0x080be5d4 in worker_thread (thd=0x9ab88c0, dummy=0x9aa8a38) at worker.c:876
#21 0xb7ede868 in dummy_worker (opaque=0x9ab88c0) at threadproc/unix/thread.c:138
#22 0x006bb341 in start_thread () from /lib/tls/libpthread.so.0
#23 0x005e36fe in clone () from /lib/tls/libc.so.6
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
rpluem@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |RESOLVED
Resolution| |FIXED
------- Additional Comments From rpluem@apache.org 2006-04-22 15:28 -------
Backported to 2.2.x as r396049 ( http://svn.apache.org/viewcvs?rev=396049&view=rev).
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
rpluem@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #17836|0 |1
is obsolete| |
------- Additional Comments From rpluem@apache.org 2006-03-12 00:12 -------
Created an attachment (id=17874)
--> (http://issues.apache.org/bugzilla/attachment.cgi?id=17874&action=view)
Patch against trunk to fix double return of connections to pool
I created a patch that removes the double calls to ap_proxy_http_cleanup and
adds a check if a connection has been already returned to the connection pool
before. Could you please give the patch a try and let me know if this fixes
your problem and if you do not see any error messages of "proxy: Pooled
connection 0x%pp for worker %s has been already returned to the connection
pool.". Thanks.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
rpluem@apache.org changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
------- Additional Comments From rpluem@apache.org 2006-03-06 23:04 -------
Many thanks for the update. It was very helpful. The problem seems to be caused
by the fact that apr_reslist_release does not notice "double releases" (see also
http://marc.theaimsgroup.com/?l=apr-dev&m=114168465029319&w=2).
I am checking now if this is a bug inside apr-util or inside the proxy code.
If it is a bug inside the proxy code, I need to implement additional measures to
prevent such "double releases" (apart from removing the unnecessary double calls
to ap_proxy_http_cleanup which you already mentioned). These additional measures
should help track down such issues faster in the future and prevent the server
from segfaulting in such situations. I'll keep you updated.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
------- Additional Comments From rpluem@apache.org 2006-04-17 10:43 -------
Proposed for backport to 2.2.x as r394653
(http://svn.apache.org/viewcvs?rev=394653&view=rev).
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
------- Additional Comments From mk-asf@gigacodes.de 2006-03-28 12:13 -------
We applied the patch about 10 days ago, no double-free warnings so far. Crash is
no longer reproducable with ab in test environment.
However we are still seeing segfaults in our logfiles in production use. I have
the suspect it might be related to mod_ssl, but I did not have time to look into
this so far.
So this patch is probably ready to roll, although there's more to do ...
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
------- Additional Comments From rpluem@apache.org 2006-04-14 13:20 -------
Committed patch to trunk as r394088
(http://svn.apache.org/viewcvs?rev=394088&view=rev).
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
------- Additional Comments From mk-asf@gigacodes.de 2006-03-06 20:35 -------
BTW: this also fixes the " remote server (null) " thing of the error line.
see also: http://issues.apache.org/bugzilla/show_bug.cgi?id=37770#c2
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
------- Additional Comments From rpluem@apache.org 2006-03-05 14:13 -------
Created an attachment (id=17836)
--> (http://issues.apache.org/bugzilla/attachment.cgi?id=17836&action=view)
Temporary debug patch to narrow down the problem.
Could you please apply the attached patch temporarily and set the LogLevel to
debug? A conn data structure should be only leased to one thread at the same
time from the connection pool. So in theory different threads should not have
access to the same data structure at the same time. The patch logs the aquire
and release operations and hopefully helps to see if this is not the case.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
------- Additional Comments From mk-asf@gigacodes.de 2006-03-04 19:18 -------
I did some more debugging and found a strange behaviour in ap_proxy_connect_backend
#0 ap_proxy_connect_backend (proxy_function=0x80d5ff9 "HTTP", conn=0x9d0cfd8,
worker=0x9c95db0, s=0x9d154c8) at proxy_util.c:2045
rv = 164679640
connected = 1
loglevel = 164190680
backend_addr = (apr_sockaddr_t *) 0x9d60a00
newsock = (apr_socket_t *) 0x9d8fa28
#1 0x0809c748 in proxy_http_handler (r=0x9d99af8, worker=0x9c95db0,
conf=0x9c959d8, url=0x9d91268 "/test/output.php?delay=0", proxyname=0x0,
proxyport=0)
at mod_proxy_http.c:1691
I put a breakpoint on the return statement in case that conn->sock is null,
however the function will return true, because its internal newsock is valid and
its connected counter is 1. (see stacktrace above). conn looks magically empty:
(gdb) inspect conn->hostname
$4 = 0x0
(gdb) inspect conn->port
$5 = 0
(gdb) inspect conn->pool
$6 = (apr_pool_t *) 0x9d60768
(gdb) inspect conn->is_ssl
$7 = 0
(gdb) inspect conn->sock
$8 = (apr_socket_t *) 0x0
(gdb) inspect conn->addr
$9 = (apr_sockaddr_t *) 0x0
(gdb) inspect conn->flags
$10 = 0
(gdb) inspect conn->close
$11 = 0
(gdb) inspect conn->close_on_recycle
$12 = 0
(gdb) inspect conn->worker
$13 = (proxy_worker *) 0x9c95db0
connected is only set right after conn->sock = newsock , both vars are never
touched until return then. How does conn become so empty ? I also do not have
any of the error messages from ap_proxy_connect_backend in the error_log, so I
assume the bad boy might be another thread.
I get the following error messages from other threads and they are already int
the logfile when the above breakpoint is hit, so chances are that things get
messed up here.
[Sat Mar 04 18:12:48 2006] [error] [client 127.0.0.1] proxy: error reading
status line from remote server (null)
[Sat Mar 04 18:12:48 2006] [error] [client 127.0.0.1] proxy: Error reading from
remote server returned by /test/output.php
above this errormessage is
ap_proxy_http_cleanup(NULL, r, backend);
which i think is redundant as proxy_http_handler runs cleanup: anyway. removing
did not fix the problem.
Let me know if this looks like something to go into deeper.
(PS. We are running mod_worker)
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 38793] - Segfault when backend failes, threading issue
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=38793>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=38793
------- Additional Comments From mk-asf@gigacodes.de 2006-03-06 20:16 -------
switch loglevel to debug eliminates problem ( probably because of a change in
timing because of the additional overhead).
I changed the patch to log as ERROR. I grepped the logfile for the segfaulting
backend and all non "backend" messages:
[Mon Mar 06 19:37:35 2006] [error] proxy: HTTP: Aquired backend 0x987bff8 for
request 0xa9e20540, connection 0xaa01ab08, pid 13549, tid 0xb4184bb0
[Mon Mar 06 19:37:35 2006] [error] proxy: HTTP: Released backend 0x987bff8 for
request 0xa9e20540, connection 0xaa01ab08, pid 13549, tid 0xb4184bb0
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: error reading
status line from remote server (null)
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: Error reading from
remote server returned by /test/output.php
[Mon Mar 06 19:37:35 2006] [error] proxy: HTTP: Released backend 0x987bff8 for
request 0xa9e20540, connection 0xaa01ab08, pid 13549, tid 0xb4184bb0
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: error reading
status line from remote server (null)
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: Error reading from
remote server returned by /test/output.php
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: error reading
status line from remote server (null)
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: Error reading from
remote server returned by /test/output.php
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: error reading
status line from remote server (null)
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: Error reading from
remote server returned by /test/output.php
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: error reading
status line from remote server (null)
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: Error reading from
remote server returned by /test/output.php
[Mon Mar 06 19:37:35 2006] [error] proxy: HTTP: Aquired backend 0x987bff8 for
request 0xaa034be8, connection 0xaa030d90, pid 13549, tid 0xadd7abb0
[Mon Mar 06 19:37:35 2006] [error] proxy: HTTP: Aquired backend 0x987bff8 for
request 0xa9e2a568, connection 0xaa018a90, pid 13549, tid 0xaf17cbb0
[Mon Mar 06 19:37:35 2006] [error] proxy: HTTP: Released backend 0x987bff8 for
request 0xa9e2a568, connection 0xaa018a90, pid 13549, tid 0xaf17cbb0
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: error reading
status line from remote server (null)
[Mon Mar 06 19:37:35 2006] [error] [client 127.0.0.1] proxy: Error reading from
remote server returned by /test/output.php
[Mon Mar 06 19:37:35 2006] [error] proxy: HTTP: Released backend 0x987bff8 for
request 0xa9e2a568, connection 0xaa018a90, pid 13549, tid 0xaf17cbb0
The backend gets released, "error reading status line" gets logged, and the same
backend gets released again. Afterwards it gets acquired by 2 different threads.
One thread fails with same error (0xaf17cbb0) releasing the backend, the other
thread gets the segfault (0xadd7abb0) while in ap_proxy_connection_create. (gdb bt).
I suspect ap_proxy_http_cleanup(NULL, r, backend) in line 1220 ( and 1246,
numbers my vary because of your patch ) of mod_proxy_http.c to be the problem,
as mentioned in previous comment, it gets called there and again in line 1741,
after the cleanup:-goto. I don't know what the proxy_function argument means in
line 1741, because the other cleanups are called with NULL ?
Removing line 1220 eliminates the Segfaults. But I get these lines after some
time running ab:
(99)Cannot assign requested address: proxy: HTTP: attempt to connect to
127.0.0.1:8081 (127.0.0.1) failed
I guess this could be related to a temporary shortage of tcp-ip source ports
(I'm running ab from the same machine and it never happens for the first 20000
requests)
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org