You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2012/02/03 20:57:33 UTC
DO NOT REPLY [Bug 52595] New: requests with gzip+chunked encoded
body don't proxy reliably
https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
Bug #: 52595
Summary: requests with gzip+chunked encoded body don't proxy
reliably
Product: Apache httpd-2
Version: 2.2.21
Platform: PC
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_deflate
AssignedTo: bugs@httpd.apache.org
ReportedBy: lmeyer@vmware.com
Classification: Unclassified
Created attachment 28258
--> https://issues.apache.org/bugzilla/attachment.cgi?id=28258
Tools for reproducing the problem - script, test load, config, backend JSP
When request bodies are posted with gzip compression + chunked transfer, then
proxied to a Tomcat backend, they are subject to sporadic failures with all
three proxy methods I've tried: mod_proxy_http, mod_proxy_ajp, and mod_jk.
The problem does not occur without gzip. The failures are similar enough that I
suspect the problem has to do with the hand-off from mod_deflate, though it
could be further along in the request handling. It does not succeed or fail the
same each request even when exactly the same content is sent with exactly the
same chunk sizes, so I suspect that network timing and/or internal state may
play a role.
This may be related to https://issues.apache.org/bugzilla/show_bug.cgi?id=52492
- then again, it may not. It seemed different enough to me to merit a separate
issue.
This problem came from a real-world system issue that initially I suspected was
an AJP problem. When transferred via mod_jk, the request body would
sporadically be truncated when it was received by the backend (as confirmed by
packet trace). Aside from that all appears fine.
With mod_proxy_ajp, this use case never works at all that I have seen. The
entire body may be transmitted, but instead of sending the finishing 0-size
chunk, httpd closes the proxy connection and returns a 200 to the client (with
empty body). Errors in the log indicate mod_proxy_ajp.c(382): (20014)Internal
error: ap_get_brigade failed
With mod_proxy_http, this manifests as a 0-size chunk coming in the middle of
the request transfer. The backend interprets this as the end of the request
(which is normally what the 0-size chunk means) and responds normally to that
truncated request. Then when the proxy continues sending more chunks, the
backend returns a 400 and closes the connection. The response to the client
depends on timing - either the 200 response from the backend is returned, or
(if the connection is closed before the request is fully sent) a 502 error is
returned.
Side note: mod_proxy_http passes along the Content-Encoding: gzip header to the
backend even though the traffic is not compressed at that point. This is
probably a bug and might cause problems with some backends.
In order to test all this, I put together a Perl script that can gzip and chunk
a request allowing me to control chunk sizes (I found that some patterns made
this occur more often than others). I also created a JSP for the backend that
counts the incoming bytes and returns the count in the response so that the
script can verify whether the full content made it through. The content is
arbitrary, but for my tests, I chose a small portion of the system dictionary.
These are all attached.
I will further attach logs and packet traces recording failures with each proxy
method. These are against 2.2.21, but I have observed the same results with
other versions (2.2.22, 2.2.17, 2.3.16). Again, these can be very sporadic, so
when reproducing, many requests may be required before anything goes wrong (the
script automates this). You may see many successes or many failures in a row.
It would not surprise me if this varied by OS, libz version, architecture, etc.
but I know it's not specific to my test system. In general, a good way to
trigger the problem is the "randub" algorithm with a small chunk size (like 100
to send chunks with random sizes between 100 and 200).
My tests have been on Linux (CentOS 5).
$ uname -a
Linux 2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:32:29 EDT 2011 i686 i686 i386
GNU/Linux
$ rpm -q zlib
zlib-1.2.3-4.el5
Thanks for taking a look at this.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 52595] requests with gzip+chunked encoded body
don't proxy reliably
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
--- Comment #3 from Luke Meyer <lm...@vmware.com> 2012-02-03 20:08:52 UTC ---
Created attachment 28261
--> https://issues.apache.org/bugzilla/attachment.cgi?id=28261
Test case and results for mod_proxy_ajp method
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 52595] requests with gzip+chunked encoded body
don't proxy reliably
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
--- Comment #2 from Luke Meyer <lm...@vmware.com> 2012-02-03 20:08:15 UTC ---
Created attachment 28260
--> https://issues.apache.org/bugzilla/attachment.cgi?id=28260
Test case and results for mod_proxy_http method
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 52595] requests with gzip+chunked encoded body
don't proxy reliably
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
--- Comment #1 from Luke Meyer <lm...@vmware.com> 2012-02-03 20:07:32 UTC ---
Created attachment 28259
--> https://issues.apache.org/bugzilla/attachment.cgi?id=28259
Test case and results for mod_jk method (only one success and failure from the
run are included in the trace, to keep size down)
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 52595] requests with gzip+chunked encoded body don't proxy
reliably
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
FN <li...@felix-nawroth.net> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |lists@felix-nawroth.net
--- Comment #5 from FN <li...@felix-nawroth.net> ---
I can confirm that mod_deflate leaves the header "Content-Encoding: gzip" in
forwarded traffic when using mod_proxy. This causes BadRequest-errors when
using a proxy chain in which every proxy is configured with InputFilter
Deflate.
The problem of sporadic BadRequest-errors is not limited to chunked requests,
we're seeing the same with normal POST-requests with gzipped body. The error
rate varies.
Tested in versions 2.2.18 to 2.2.22. I'll be happy to provide additional
information as needed.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 52595] requests with gzip+chunked encoded body
don't proxy reliably
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
--- Comment #4 from Luke Meyer <lm...@vmware.com> 2012-02-03 21:31:12 UTC ---
Might help if I add that in all tests, httpd was listening at port 8000, Tomcat
was listening on the same host at 8080 for http and 8009 for ajp.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 52595] requests with gzip+chunked encoded body don't proxy
reliably
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=52595
mik4@gmx.ch changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |mik4@gmx.ch
--- Comment #6 from mik4@gmx.ch ---
This issue is probably caused by bug 54255.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org