You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2013/10/09 21:46:44 UTC
[Bug 55643] New: truncated CGI output
https://issues.apache.org/bugzilla/show_bug.cgi?id=55643
Bug ID: 55643
Summary: truncated CGI output
Product: Apache httpd-2
Version: 2.4.6
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: Core
Assignee: bugs@httpd.apache.org
Reporter: utoddl@email.unc.edu
Under some circumstances, CGI output will be truncated. The bug is in
server/core_filters.c, around line 542 in the function ap_core_output_filter().
No, really!
The call to send_brigade_blocking() is expected to return APR_SUCCESS,
otherwise the connection to the browser is assumed abandoned. However,
send_brigade_blocking() calls apr_poll(), which calls the system poll(). Under
extremely hard to reproduce on demand circumstances, the poll() call is
interrupted by the SIGCHLD signal from the exiting CGI script, setting errno to
EINTR ('4' on Linux). This value propagates up through apr_poll() and
send_brigade_blocking() into ap_core_output_filter()'s "rv", where the rest of
the output is abandoned, like so:
rv = send_brigade_blocking(net->client_socket, bb,
&(ctx->bytes_written), c);
if (rv != APR_SUCCESS) {
/* The client has aborted the connection */
c->aborted = 1;
return rv;
}
I'll defer to the httpd devs whether this EINTR should be handled at this level
or further down in the stack.
As difficult as this bug is to trigger on purpose, it started showing up
consistently after we upgraded our www cluster (4 RHEL servers) to httpd-2.4.6
in early September 2013, regular enough that we were able to start debugging on
our production hosts. However, it is extremely sensitive to load, network
latency, etc., as you have to get the SIGCHLD signal from the CGI process to
come in during the poll() -- not a very big target window on modern machines
and networks.
Searches turned up dozens of reports over the years that could well be this
bug. In most cases people fiddled with things until the problem went into
hiding, at which point (not surprisingly) they quit working on it. We were
lucky enough that it just wouldn't go away on our systems.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 55643] truncated CGI output
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=55643
--- Comment #1 from Jeff Trawick <tr...@apache.org> ---
EINTR should be handled at the point of the poll... (On Unix, some callers
have the opportunity to handle EINTR differently when apr_poll() is
interrupted.) Did you try something like this?
Index: server/core_filters.c
===================================================================
--- server/core_filters.c (revision 1530310)
+++ server/core_filters.c (working copy)
@@ -745,7 +745,9 @@
pollset.reqevents = APR_POLLOUT;
pollset.desc.s = s;
apr_socket_timeout_get(s, &timeout);
- rv = apr_poll(&pollset, 1, &nsds, timeout);
+ do {
+ rv = apr_poll(&pollset, 1, &nsds, timeout);
+ } while (APR_STATUS_IS_EINTR(rv));
if (rv != APR_SUCCESS) {
break;
}
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 55643] truncated CGI output
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=55643
--- Comment #2 from Todd Lewis <ut...@email.unc.edu> ---
(In reply to Jeff Trawick from comment #1)
That does indeed seem to fix the problem. Before, strace clearly showed
poll([{fd=17, events=POLLOUT}], 1, 200000) = -1 EINTR (Interrupted system call)
and the rest of the writev()s never happened. Now I'm seeing
poll([{fd=17, events=POLLOUT}], 1, 200000) = -1 EINTR (Interrupted system call)
--- SIGCHLD (Child exited) @ 0 (0) ---
poll([{fd=17, events=POLLOUT}], 1, 200000) = 1 ([{fd=17, revents=POLLOUT}])
followed by more writev()s.
Thanks very much.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 55643] truncated CGI output
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=55643
Jeff Trawick <tr...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution|--- |FIXED
--- Comment #4 from Jeff Trawick <tr...@apache.org> ---
This is now in the 2.4.x branch (r1530999) for the next 2.4 release.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 55643] truncated CGI output
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=55643
--- Comment #3 from Jeff Trawick <tr...@apache.org> ---
Thanks for the quick feedback... The same change is committed to trunk as
r1530793. I'll propose backporting the fix to the 2.4.x branch after I test a
little more (though this is about as simple as it gets).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 55643] truncated CGI output
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=55643
Jeff Trawick <tr...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |CLOSED
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org