You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Doug Herbert <do...@yahoo.com> on 2010/04/09 05:09:32 UTC

Fragmented delivery of servlet request

Thoughts welcomed on the following problem :

Centos 5.4, http 2.2.3, tomcat5-5.5.23-0jpp.7.el5_3.2, java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5


A large servlet request, POST'ed from client to apache, connected using proxy_ajp to ajp://localhost:8009.

reassembled on server eth0 as 10766 bytes but only partially sent to tomcat. Th eth0 line trace to http ( port 80 ) was split over 19 packets of 536 bytes each.

My initial thoughts were, that maxHttpHeaderSize=8192, was too low. Increasing to 16384 did not resolve the issue.

So more wireshark line traces, ( one tcpdump across eth0 capturing the http POST and a 2nd tcpdump across loopback capturing ajp connector traffic ) revealed, that apache via connector ajp delivered each packet realtime time to tomcat, without waiting for all 10766 bytes to arrive, though the trace across loopback on port 8009, revealed that tomcat starting the reply before all 10766 bytes had arrived. 

tcpdump on eth0 confirmed, by reassembled tcp segment to contained the 10766 bytes from the browser client.

( Note : I have mangled URI SRV referer host headers )

Apache JServ Protocol v1.3
    Magic: 1234
    Length: 528
    Code: (2) FORWARD REQUEST
    Method: (4) POST
    Version: HTTP/1.1
    URI: /xxxx/yyyyy
    RADDR: 192.168.252.68
    RHOST: 
    SRV: xxxyyyy
    PORT: 80
    SSLP: 0
    NHDR: 11
    accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */*
    referer: http://xxxxyyyy
    accept-language: en-us
    content-type: application/x-www-form-urlencoded
    UA-CPU: x86
    accept-encoding: gzip, deflate
    user-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
    host: xxxyyyy
    content-length: 10766
    connection: Keep-Alive
    Cache-Control: no-cache


My question is, why did tomcat start to send the RSP ( response ) after only receiving the first data packet from ajp connector, around 500 bytes, when the above states the content length is 10766 bytes. You can also see that subsequent REQ ( request body ) data packets are still transmitted to tomcat, though the response is already being returned. 

Another option I thought of, was to buffer the whole 10766 bytes up on the apache side, and then get the connector to pass the request across as one 'big' packet. Is this possible to configure ? If so, then maxHttpHeaderSize will come into play, though at the moment only many small packets are being sent across in the fragmented request.

A stack dump in catalina.out can be seen below, where the input filter is balking on the POSTed parameters, most likely because only 550 bytes of the 10766, have turned up for the input filter to process.

Apr 9, 2010 10:51:55 AM org.apache.catalina.connector.Request parseParameters
WARNING: Exception thrown whilst processing POSTed parameters
java.io.IOException: Socket read failed
        at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1038)
        at org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1159)
        at org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1091)
        at org.apache.coyote.ajp.AjpAprProcessor.refillReadBuffer(AjpAprProcessor.java:1130)
        at org.apache.coyote.ajp.AjpAprProcessor.access$0(AjpAprProcessor.java:1115)
        at org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1233)
        at org.apache.coyote.Request.doRead(Request.java:419)
        at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265)
        at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
        at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280)
        at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
        at org.apache.catalina.connector.Request.readPostBody(Request.java:2419)
        at org.apache.catalina.connector.Request.parseParameters(Request.java:2398)
        at org.apache.catalina.connector.Request.getParameter(Request.java:1005)
        at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:353)
        at com.xxxx.yyyyy.filters.RequestValidation.getParameter(RequestValidation.java:40)
        at com.xxxx.yyyy.zzzzz.service(zzzzz.java:157)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at com.xxxx.yyyy.filters.RequestFilter.doFilter(RequestFilter.java:16)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
        at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:444)





No.     Time            Source                SrcPort Destination           DPort  Protocol Info
    731 10:24:51.406434 127.0.0.1             46704   127.0.0.1             8009   AJP13    0:REQ:POST /xxxx/yyyy HTTP/1.1
    732 10:24:51.406452 127.0.0.1             46704   127.0.0.1             8009   AJP13    0:REQ:Body
    733 10:24:51.408554 127.0.0.1             8009    127.0.0.1             46704  TCP      8009 > 46704 [ACK] Seq=273919 Ack=8041 Win=386 Len=0 TSV=192177833 TSER=192177830
    734 10:24:51.408730 127.0.0.1             8009    127.0.0.1             46704  AJP13    0:RSP:GET BODY CHUNK
    735 10:24:51.408737 127.0.0.1             46704   127.0.0.1             8009   TCP      46704 > 8009 [ACK] Seq=8041 Ack=273926 Win=2440 Len=0 TSV=192177833 TSER=192177833
    742 10:24:51.460670 127.0.0.1             46704   127.0.0.1             8009   AJP13    0:REQ:Body
    743 10:24:51.460728 127.0.0.1             8009    127.0.0.1             46704  AJP13    0:RSP:GET BODY CHUNK
    767 10:24:51.500813 127.0.0.1             46704   127.0.0.1             8009   TCP      46704 > 8009 [ACK] Seq=8583 Ack=273933 Win=2441 Len=0 TSV=192177925 TSER=192177885
    768 10:24:51.562298 127.0.0.1             8009    127.0.0.1             46704  AJP13    0:RSP:SEND BODY CHUNK
    769 10:24:51.562316 127.0.0.1             46704   127.0.0.1             8009   TCP      46704 > 8009 [ACK] Seq=8583 Ack=282304 Win=2389 Len=0 TSV=192177986 TSER=192177986
    770 10:24:51.562662 127.0.0.1             8009    127.0.0.1             46704  AJP13    0:RSP:SEND BODY CHUNK
    771 10:24:51.562670 127.0.0.1             46704   127.0.0.1             8009   TCP      46704 > 8009 [ACK] Seq=8583 Ack=298688 Win=2291 Len=0 TSV=192177987 TSER=192177987
    772 10:24:51.581234 127.0.0.1             8009    127.0.0.1             46704  AJP13    0:RSP:END RESPONSE
    773 10:24:51.581243 127.0.0.1             46704   127.0.0.1             8009   TCP      46704 > 8009 [ACK] Seq=8583 Ack=307526 Win=2236 Len=0 TSV=192178005 TSER=192178005
    783 10:24:51.700763 127.0.0.1             46704   127.0.0.1             8009   AJP13    0:REQ:Body
    784 10:24:51.701186 127.0.0.1             8009    127.0.0.1             46704  AJP13    0:RSP:END RESPONSE
    785 10:24:51.701194 127.0.0.1             46704   127.0.0.1             8009   TCP      46704 > 8009 [ACK] Seq=9125 Ack=307555 Win=2417 Len=0 TSV=192178125 TSER=192178125




      

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Fragmented delivery of servlet request

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Doug,

On 4/9/2010 2:31 PM, Doug Herbert wrote:
> some timimgs :
> 
> took 1500 milli seconds for all 19 packets to arrive at that web server, from the client's browser connected across a 'slow' cellular network.
> 
> took 2 ( two ) milli seconds for tomcat to balk at the incoming request and send back the RSP packet, after the first REQ packet sent into tomcat.
> 
> eg. in plain terms tomcat started responding before all fragmented  packets had even arrived at the web server !

Can you post your mod_proxy_ajp configuration? 2ms is a very short
amount of time for the AJP listener to reject a request because it is
incomplete, but I suppose it's possible.

I see you're running httpd 2.2.3, which is probably a package-managed
version from CentOS. Is that really 2.2.3, or does it have some
additional patches on top of it? I ask because mod_proxy_ajp has
improved considerably since 2.2.3 (the current version is 2.2.15).

Would it be possible for you to test against a more recent version, even
if you can't upgrade in production quite yet? If this is not possible,
how about switching (at least temporarily) to mod_jk, compiled yourself?
It's possible that something in the last (quite a) few versions has been
identified and fixed. Is that's the case, there's no reason to chase our
tails re-identifying a bug that's already been fixed.

If you have gcc and apxs handy, compiling mod_jk is super easy, as is
installation and configuration.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAku/hJ4ACgkQ9CaO5/Lv0PBbsQCghjkyTMumm8eQkJ63YS5sHYUo
dMoAn3alHa51nNh0DvtWCY6PrMPavLpB
=ZFu9
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Fragmented delivery of servlet request

Posted by Doug Herbert <do...@yahoo.com>.
some timimgs :

took 1500 milli seconds for all 19 packets to arrive at that web server, from the client's browser connected across a 'slow' cellular network.

took 2 ( two ) milli seconds for tomcat to balk at the incoming request and send back the RSP packet, after the first REQ packet sent into tomcat.

eg. in plain terms tomcat started responding before all fragmented  packets had even arrived at the web server !

--- On Fri, 9/4/10, Doug Herbert <do...@yahoo.com> wrote:

> From: Doug Herbert <do...@yahoo.com>
> Subject: Fragmented delivery of servlet request
> To: users@tomcat.apache.org
> Received: Friday, 9 April, 2010, 3:09 PM
> Thoughts welcomed on the following
> problem :
> 
> Centos 5.4, http 2.2.3, tomcat5-5.5.23-0jpp.7.el5_3.2,
> java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5
> 
> 
> A large servlet request, POST'ed from client to apache,
> connected using proxy_ajp to ajp://localhost:8009.
> 
> reassembled on server eth0 as 10766 bytes but only
> partially sent to tomcat. Th eth0 line trace to http ( port
> 80 ) was split over 19 packets of 536 bytes each.
> 
> My initial thoughts were, that maxHttpHeaderSize=8192, was
> too low. Increasing to 16384 did not resolve the issue.
> 
> So more wireshark line traces, ( one tcpdump across eth0
> capturing the http POST and a 2nd tcpdump across loopback
> capturing ajp connector traffic ) revealed, that apache via
> connector ajp delivered each packet realtime time to tomcat,
> without waiting for all 10766 bytes to arrive, though the
> trace across loopback on port 8009, revealed that tomcat
> starting the reply before all 10766 bytes had arrived. 
> 
> tcpdump on eth0 confirmed, by reassembled tcp segment to
> contained the 10766 bytes from the browser client.
> 
> ( Note : I have mangled URI SRV referer host headers )
> 
> Apache JServ Protocol v1.3
>     Magic: 1234
>     Length: 528
>     Code: (2) FORWARD REQUEST
>     Method: (4) POST
>     Version: HTTP/1.1
>     URI: /xxxx/yyyyy
>     RADDR: 192.168.252.68
>     RHOST: 
>     SRV: xxxyyyy
>     PORT: 80
>     SSLP: 0
>     NHDR: 11
>     accept: image/gif, image/x-xbitmap,
> image/jpeg, image/pjpeg, application/x-shockwave-flash,
> application/vnd.ms-powerpoint, application/vnd.ms-excel,
> application/msword, */*
>     referer: http://xxxxyyyy
>     accept-language: en-us
>     content-type:
> application/x-www-form-urlencoded
>     UA-CPU: x86
>     accept-encoding: gzip, deflate
>     user-agent: Mozilla/4.0 (compatible; MSIE
> 7.0; Windows NT 5.1)
>     host: xxxyyyy
>     content-length: 10766
>     connection: Keep-Alive
>     Cache-Control: no-cache
> 
> 
> My question is, why did tomcat start to send the RSP (
> response ) after only receiving the first data packet from
> ajp connector, around 500 bytes, when the above states the
> content length is 10766 bytes. You can also see that
> subsequent REQ ( request body ) data packets are still
> transmitted to tomcat, though the response is already being
> returned. 
> 
> Another option I thought of, was to buffer the whole 10766
> bytes up on the apache side, and then get the connector to
> pass the request across as one 'big' packet. Is this
> possible to configure ? If so, then maxHttpHeaderSize will
> come into play, though at the moment only many small packets
> are being sent across in the fragmented request.
> 
> A stack dump in catalina.out can be seen below, where the
> input filter is balking on the POSTed parameters, most
> likely because only 550 bytes of the 10766, have turned up
> for the input filter to process.
> 
> Apr 9, 2010 10:51:55 AM
> org.apache.catalina.connector.Request parseParameters
> WARNING: Exception thrown whilst processing POSTed
> parameters
> java.io.IOException: Socket read failed
>         at
> org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1038)
>         at
> org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1159)
>         at
> org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1091)
>         at
> org.apache.coyote.ajp.AjpAprProcessor.refillReadBuffer(AjpAprProcessor.java:1130)
>         at
> org.apache.coyote.ajp.AjpAprProcessor.access$0(AjpAprProcessor.java:1115)
>         at
> org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1233)
>         at
> org.apache.coyote.Request.doRead(Request.java:419)
>         at
> org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265)
>         at
> org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
>         at
> org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280)
>         at
> org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
>         at
> org.apache.catalina.connector.Request.readPostBody(Request.java:2419)
>         at
> org.apache.catalina.connector.Request.parseParameters(Request.java:2398)
>         at
> org.apache.catalina.connector.Request.getParameter(Request.java:1005)
>         at
> org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:353)
>         at
> com.xxxx.yyyyy.filters.RequestValidation.getParameter(RequestValidation.java:40)
>         at
> com.xxxx.yyyy.zzzzz.service(zzzzz.java:157)
>         at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>         at
> com.xxxx.yyyy.filters.RequestFilter.doFilter(RequestFilter.java:16)
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
>         at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
>         at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
>         at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>         at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
>         at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
>         at
> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:444)
> 
> 
> 
> 
> 
> No.     Time       
>     Source         
>       SrcPort Destination     
>      DPort  Protocol Info
>     731 10:24:51.406434 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   AJP13   
> 0:REQ:POST /xxxx/yyyy HTTP/1.1
>     732 10:24:51.406452 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   AJP13   
> 0:REQ:Body
>     733 10:24:51.408554 127.0.0.1   
>          8009   
> 127.0.0.1         
>    46704  TCP      8009
> > 46704 [ACK] Seq=273919 Ack=8041 Win=386 Len=0
> TSV=192177833 TSER=192177830
>     734 10:24:51.408730 127.0.0.1   
>          8009   
> 127.0.0.1         
>    46704  AJP13    0:RSP:GET
> BODY CHUNK
>     735 10:24:51.408737 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   TCP   
>   46704 > 8009 [ACK] Seq=8041 Ack=273926 Win=2440
> Len=0 TSV=192177833 TSER=192177833
>     742 10:24:51.460670 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   AJP13   
> 0:REQ:Body
>     743 10:24:51.460728 127.0.0.1   
>          8009   
> 127.0.0.1         
>    46704  AJP13    0:RSP:GET
> BODY CHUNK
>     767 10:24:51.500813 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   TCP   
>   46704 > 8009 [ACK] Seq=8583 Ack=273933 Win=2441
> Len=0 TSV=192177925 TSER=192177885
>     768 10:24:51.562298 127.0.0.1   
>          8009   
> 127.0.0.1         
>    46704  AJP13    0:RSP:SEND
> BODY CHUNK
>     769 10:24:51.562316 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   TCP   
>   46704 > 8009 [ACK] Seq=8583 Ack=282304 Win=2389
> Len=0 TSV=192177986 TSER=192177986
>     770 10:24:51.562662 127.0.0.1   
>          8009   
> 127.0.0.1         
>    46704  AJP13    0:RSP:SEND
> BODY CHUNK
>     771 10:24:51.562670 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   TCP   
>   46704 > 8009 [ACK] Seq=8583 Ack=298688 Win=2291
> Len=0 TSV=192177987 TSER=192177987
>     772 10:24:51.581234 127.0.0.1   
>          8009   
> 127.0.0.1         
>    46704  AJP13    0:RSP:END
> RESPONSE
>     773 10:24:51.581243 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   TCP   
>   46704 > 8009 [ACK] Seq=8583 Ack=307526 Win=2236
> Len=0 TSV=192178005 TSER=192178005
>     783 10:24:51.700763 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   AJP13   
> 0:REQ:Body
>     784 10:24:51.701186 127.0.0.1   
>          8009   
> 127.0.0.1         
>    46704  AJP13    0:RSP:END
> RESPONSE
>     785 10:24:51.701194 127.0.0.1   
>      
>    46704   127.0.0.1 
>        
>    8009   TCP   
>   46704 > 8009 [ACK] Seq=9125 Ack=307555 Win=2417
> Len=0 TSV=192178125 TSER=192178125
> 
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 
> 


      

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org