You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Doug Herbert <do...@yahoo.com> on 2010/04/09 05:09:32 UTC
Fragmented delivery of servlet request
Thoughts welcomed on the following problem :
Centos 5.4, http 2.2.3, tomcat5-5.5.23-0jpp.7.el5_3.2, java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5
A large servlet request, POST'ed from client to apache, connected using proxy_ajp to ajp://localhost:8009.
reassembled on server eth0 as 10766 bytes but only partially sent to tomcat. Th eth0 line trace to http ( port 80 ) was split over 19 packets of 536 bytes each.
My initial thoughts were, that maxHttpHeaderSize=8192, was too low. Increasing to 16384 did not resolve the issue.
So more wireshark line traces, ( one tcpdump across eth0 capturing the http POST and a 2nd tcpdump across loopback capturing ajp connector traffic ) revealed, that apache via connector ajp delivered each packet realtime time to tomcat, without waiting for all 10766 bytes to arrive, though the trace across loopback on port 8009, revealed that tomcat starting the reply before all 10766 bytes had arrived.
tcpdump on eth0 confirmed, by reassembled tcp segment to contained the 10766 bytes from the browser client.
( Note : I have mangled URI SRV referer host headers )
Apache JServ Protocol v1.3
Magic: 1234
Length: 528
Code: (2) FORWARD REQUEST
Method: (4) POST
Version: HTTP/1.1
URI: /xxxx/yyyyy
RADDR: 192.168.252.68
RHOST:
SRV: xxxyyyy
PORT: 80
SSLP: 0
NHDR: 11
accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */*
referer: http://xxxxyyyy
accept-language: en-us
content-type: application/x-www-form-urlencoded
UA-CPU: x86
accept-encoding: gzip, deflate
user-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
host: xxxyyyy
content-length: 10766
connection: Keep-Alive
Cache-Control: no-cache
My question is, why did tomcat start to send the RSP ( response ) after only receiving the first data packet from ajp connector, around 500 bytes, when the above states the content length is 10766 bytes. You can also see that subsequent REQ ( request body ) data packets are still transmitted to tomcat, though the response is already being returned.
Another option I thought of, was to buffer the whole 10766 bytes up on the apache side, and then get the connector to pass the request across as one 'big' packet. Is this possible to configure ? If so, then maxHttpHeaderSize will come into play, though at the moment only many small packets are being sent across in the fragmented request.
A stack dump in catalina.out can be seen below, where the input filter is balking on the POSTed parameters, most likely because only 550 bytes of the 10766, have turned up for the input filter to process.
Apr 9, 2010 10:51:55 AM org.apache.catalina.connector.Request parseParameters
WARNING: Exception thrown whilst processing POSTed parameters
java.io.IOException: Socket read failed
at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1038)
at org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1159)
at org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1091)
at org.apache.coyote.ajp.AjpAprProcessor.refillReadBuffer(AjpAprProcessor.java:1130)
at org.apache.coyote.ajp.AjpAprProcessor.access$0(AjpAprProcessor.java:1115)
at org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1233)
at org.apache.coyote.Request.doRead(Request.java:419)
at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265)
at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280)
at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
at org.apache.catalina.connector.Request.readPostBody(Request.java:2419)
at org.apache.catalina.connector.Request.parseParameters(Request.java:2398)
at org.apache.catalina.connector.Request.getParameter(Request.java:1005)
at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:353)
at com.xxxx.yyyyy.filters.RequestValidation.getParameter(RequestValidation.java:40)
at com.xxxx.yyyy.zzzzz.service(zzzzz.java:157)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at com.xxxx.yyyy.filters.RequestFilter.doFilter(RequestFilter.java:16)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:444)
No. Time Source SrcPort Destination DPort Protocol Info
731 10:24:51.406434 127.0.0.1 46704 127.0.0.1 8009 AJP13 0:REQ:POST /xxxx/yyyy HTTP/1.1
732 10:24:51.406452 127.0.0.1 46704 127.0.0.1 8009 AJP13 0:REQ:Body
733 10:24:51.408554 127.0.0.1 8009 127.0.0.1 46704 TCP 8009 > 46704 [ACK] Seq=273919 Ack=8041 Win=386 Len=0 TSV=192177833 TSER=192177830
734 10:24:51.408730 127.0.0.1 8009 127.0.0.1 46704 AJP13 0:RSP:GET BODY CHUNK
735 10:24:51.408737 127.0.0.1 46704 127.0.0.1 8009 TCP 46704 > 8009 [ACK] Seq=8041 Ack=273926 Win=2440 Len=0 TSV=192177833 TSER=192177833
742 10:24:51.460670 127.0.0.1 46704 127.0.0.1 8009 AJP13 0:REQ:Body
743 10:24:51.460728 127.0.0.1 8009 127.0.0.1 46704 AJP13 0:RSP:GET BODY CHUNK
767 10:24:51.500813 127.0.0.1 46704 127.0.0.1 8009 TCP 46704 > 8009 [ACK] Seq=8583 Ack=273933 Win=2441 Len=0 TSV=192177925 TSER=192177885
768 10:24:51.562298 127.0.0.1 8009 127.0.0.1 46704 AJP13 0:RSP:SEND BODY CHUNK
769 10:24:51.562316 127.0.0.1 46704 127.0.0.1 8009 TCP 46704 > 8009 [ACK] Seq=8583 Ack=282304 Win=2389 Len=0 TSV=192177986 TSER=192177986
770 10:24:51.562662 127.0.0.1 8009 127.0.0.1 46704 AJP13 0:RSP:SEND BODY CHUNK
771 10:24:51.562670 127.0.0.1 46704 127.0.0.1 8009 TCP 46704 > 8009 [ACK] Seq=8583 Ack=298688 Win=2291 Len=0 TSV=192177987 TSER=192177987
772 10:24:51.581234 127.0.0.1 8009 127.0.0.1 46704 AJP13 0:RSP:END RESPONSE
773 10:24:51.581243 127.0.0.1 46704 127.0.0.1 8009 TCP 46704 > 8009 [ACK] Seq=8583 Ack=307526 Win=2236 Len=0 TSV=192178005 TSER=192178005
783 10:24:51.700763 127.0.0.1 46704 127.0.0.1 8009 AJP13 0:REQ:Body
784 10:24:51.701186 127.0.0.1 8009 127.0.0.1 46704 AJP13 0:RSP:END RESPONSE
785 10:24:51.701194 127.0.0.1 46704 127.0.0.1 8009 TCP 46704 > 8009 [ACK] Seq=9125 Ack=307555 Win=2417 Len=0 TSV=192178125 TSER=192178125
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Fragmented delivery of servlet request
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Doug,
On 4/9/2010 2:31 PM, Doug Herbert wrote:
> some timimgs :
>
> took 1500 milli seconds for all 19 packets to arrive at that web server, from the client's browser connected across a 'slow' cellular network.
>
> took 2 ( two ) milli seconds for tomcat to balk at the incoming request and send back the RSP packet, after the first REQ packet sent into tomcat.
>
> eg. in plain terms tomcat started responding before all fragmented packets had even arrived at the web server !
Can you post your mod_proxy_ajp configuration? 2ms is a very short
amount of time for the AJP listener to reject a request because it is
incomplete, but I suppose it's possible.
I see you're running httpd 2.2.3, which is probably a package-managed
version from CentOS. Is that really 2.2.3, or does it have some
additional patches on top of it? I ask because mod_proxy_ajp has
improved considerably since 2.2.3 (the current version is 2.2.15).
Would it be possible for you to test against a more recent version, even
if you can't upgrade in production quite yet? If this is not possible,
how about switching (at least temporarily) to mod_jk, compiled yourself?
It's possible that something in the last (quite a) few versions has been
identified and fixed. Is that's the case, there's no reason to chase our
tails re-identifying a bug that's already been fixed.
If you have gcc and apxs handy, compiling mod_jk is super easy, as is
installation and configuration.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAku/hJ4ACgkQ9CaO5/Lv0PBbsQCghjkyTMumm8eQkJ63YS5sHYUo
dMoAn3alHa51nNh0DvtWCY6PrMPavLpB
=ZFu9
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Fragmented delivery of servlet request
Posted by Doug Herbert <do...@yahoo.com>.
some timimgs :
took 1500 milli seconds for all 19 packets to arrive at that web server, from the client's browser connected across a 'slow' cellular network.
took 2 ( two ) milli seconds for tomcat to balk at the incoming request and send back the RSP packet, after the first REQ packet sent into tomcat.
eg. in plain terms tomcat started responding before all fragmented packets had even arrived at the web server !
--- On Fri, 9/4/10, Doug Herbert <do...@yahoo.com> wrote:
> From: Doug Herbert <do...@yahoo.com>
> Subject: Fragmented delivery of servlet request
> To: users@tomcat.apache.org
> Received: Friday, 9 April, 2010, 3:09 PM
> Thoughts welcomed on the following
> problem :
>
> Centos 5.4, http 2.2.3, tomcat5-5.5.23-0jpp.7.el5_3.2,
> java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5
>
>
> A large servlet request, POST'ed from client to apache,
> connected using proxy_ajp to ajp://localhost:8009.
>
> reassembled on server eth0 as 10766 bytes but only
> partially sent to tomcat. Th eth0 line trace to http ( port
> 80 ) was split over 19 packets of 536 bytes each.
>
> My initial thoughts were, that maxHttpHeaderSize=8192, was
> too low. Increasing to 16384 did not resolve the issue.
>
> So more wireshark line traces, ( one tcpdump across eth0
> capturing the http POST and a 2nd tcpdump across loopback
> capturing ajp connector traffic ) revealed, that apache via
> connector ajp delivered each packet realtime time to tomcat,
> without waiting for all 10766 bytes to arrive, though the
> trace across loopback on port 8009, revealed that tomcat
> starting the reply before all 10766 bytes had arrived.
>
> tcpdump on eth0 confirmed, by reassembled tcp segment to
> contained the 10766 bytes from the browser client.
>
> ( Note : I have mangled URI SRV referer host headers )
>
> Apache JServ Protocol v1.3
> Magic: 1234
> Length: 528
> Code: (2) FORWARD REQUEST
> Method: (4) POST
> Version: HTTP/1.1
> URI: /xxxx/yyyyy
> RADDR: 192.168.252.68
> RHOST:
> SRV: xxxyyyy
> PORT: 80
> SSLP: 0
> NHDR: 11
> accept: image/gif, image/x-xbitmap,
> image/jpeg, image/pjpeg, application/x-shockwave-flash,
> application/vnd.ms-powerpoint, application/vnd.ms-excel,
> application/msword, */*
> referer: http://xxxxyyyy
> accept-language: en-us
> content-type:
> application/x-www-form-urlencoded
> UA-CPU: x86
> accept-encoding: gzip, deflate
> user-agent: Mozilla/4.0 (compatible; MSIE
> 7.0; Windows NT 5.1)
> host: xxxyyyy
> content-length: 10766
> connection: Keep-Alive
> Cache-Control: no-cache
>
>
> My question is, why did tomcat start to send the RSP (
> response ) after only receiving the first data packet from
> ajp connector, around 500 bytes, when the above states the
> content length is 10766 bytes. You can also see that
> subsequent REQ ( request body ) data packets are still
> transmitted to tomcat, though the response is already being
> returned.
>
> Another option I thought of, was to buffer the whole 10766
> bytes up on the apache side, and then get the connector to
> pass the request across as one 'big' packet. Is this
> possible to configure ? If so, then maxHttpHeaderSize will
> come into play, though at the moment only many small packets
> are being sent across in the fragmented request.
>
> A stack dump in catalina.out can be seen below, where the
> input filter is balking on the POSTed parameters, most
> likely because only 550 bytes of the 10766, have turned up
> for the input filter to process.
>
> Apr 9, 2010 10:51:55 AM
> org.apache.catalina.connector.Request parseParameters
> WARNING: Exception thrown whilst processing POSTed
> parameters
> java.io.IOException: Socket read failed
> at
> org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1038)
> at
> org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1159)
> at
> org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1091)
> at
> org.apache.coyote.ajp.AjpAprProcessor.refillReadBuffer(AjpAprProcessor.java:1130)
> at
> org.apache.coyote.ajp.AjpAprProcessor.access$0(AjpAprProcessor.java:1115)
> at
> org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1233)
> at
> org.apache.coyote.Request.doRead(Request.java:419)
> at
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265)
> at
> org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
> at
> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280)
> at
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
> at
> org.apache.catalina.connector.Request.readPostBody(Request.java:2419)
> at
> org.apache.catalina.connector.Request.parseParameters(Request.java:2398)
> at
> org.apache.catalina.connector.Request.getParameter(Request.java:1005)
> at
> org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:353)
> at
> com.xxxx.yyyyy.filters.RequestValidation.getParameter(RequestValidation.java:40)
> at
> com.xxxx.yyyy.zzzzz.service(zzzzz.java:157)
> at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at
> com.xxxx.yyyy.filters.RequestFilter.doFilter(RequestFilter.java:16)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
> at
> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:444)
>
>
>
>
>
> No. Time
> Source
> SrcPort Destination
> DPort Protocol Info
> 731 10:24:51.406434 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 AJP13
> 0:REQ:POST /xxxx/yyyy HTTP/1.1
> 732 10:24:51.406452 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 AJP13
> 0:REQ:Body
> 733 10:24:51.408554 127.0.0.1
> 8009
> 127.0.0.1
> 46704 TCP 8009
> > 46704 [ACK] Seq=273919 Ack=8041 Win=386 Len=0
> TSV=192177833 TSER=192177830
> 734 10:24:51.408730 127.0.0.1
> 8009
> 127.0.0.1
> 46704 AJP13 0:RSP:GET
> BODY CHUNK
> 735 10:24:51.408737 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 TCP
> 46704 > 8009 [ACK] Seq=8041 Ack=273926 Win=2440
> Len=0 TSV=192177833 TSER=192177833
> 742 10:24:51.460670 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 AJP13
> 0:REQ:Body
> 743 10:24:51.460728 127.0.0.1
> 8009
> 127.0.0.1
> 46704 AJP13 0:RSP:GET
> BODY CHUNK
> 767 10:24:51.500813 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 TCP
> 46704 > 8009 [ACK] Seq=8583 Ack=273933 Win=2441
> Len=0 TSV=192177925 TSER=192177885
> 768 10:24:51.562298 127.0.0.1
> 8009
> 127.0.0.1
> 46704 AJP13 0:RSP:SEND
> BODY CHUNK
> 769 10:24:51.562316 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 TCP
> 46704 > 8009 [ACK] Seq=8583 Ack=282304 Win=2389
> Len=0 TSV=192177986 TSER=192177986
> 770 10:24:51.562662 127.0.0.1
> 8009
> 127.0.0.1
> 46704 AJP13 0:RSP:SEND
> BODY CHUNK
> 771 10:24:51.562670 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 TCP
> 46704 > 8009 [ACK] Seq=8583 Ack=298688 Win=2291
> Len=0 TSV=192177987 TSER=192177987
> 772 10:24:51.581234 127.0.0.1
> 8009
> 127.0.0.1
> 46704 AJP13 0:RSP:END
> RESPONSE
> 773 10:24:51.581243 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 TCP
> 46704 > 8009 [ACK] Seq=8583 Ack=307526 Win=2236
> Len=0 TSV=192178005 TSER=192178005
> 783 10:24:51.700763 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 AJP13
> 0:REQ:Body
> 784 10:24:51.701186 127.0.0.1
> 8009
> 127.0.0.1
> 46704 AJP13 0:RSP:END
> RESPONSE
> 785 10:24:51.701194 127.0.0.1
>
> 46704 127.0.0.1
>
> 8009 TCP
> 46704 > 8009 [ACK] Seq=9125 Ack=307555 Win=2417
> Len=0 TSV=192178125 TSER=192178125
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org