You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Thomas Boniface <th...@stickyads.tv> on 2015/03/26 17:16:32 UTC
Tomcat 7 upgrade from 7.0.28-4 to 7.0.56-1 causes application freeze
Following the update of the tomcat 7 version from 7.0.28-4 to 7.0.56-1
hoping to improve our application performances we experienced a quick
freeze of the application under a standard load (that was handle by
previous tomcat version without particular issue).
Our application is behind a nginx server and receives a high volume of HTTP
requests from client and is using servlet 3.0 async features.
While investigating we made following observations:
1/ catalina.out shows a high number of IOException Broken pipe.
2/ a high number of CLOSE_WAIT appear between nginx and tomcat.
I have uploaded a catalina log, an lsof and a netstat capture made while
the issue was happening:
http://s000.tinyupload.com/index.php?file_id=85064286189504330545 (email
size limit prevented me from attaching them to the message).
Any pointer is welcomed. Also is there a way to have more details on
connection status of tomcat ?
Thanks,
Thomas
Re: Tomcat 7 upgrade from 7.0.28-4 to 7.0.56-1 causes application freeze
Posted by Thomas Boniface <th...@stickyads.tv>.
Hi Christopher,
Here are responses to your questions:
No thread dump were made during when the issue occured unfortunetly.
We use an Nio connector:
<Connector port="8080" protocol="org.apache.coyote.http11.Http11NioProtocol"
selectorTimeout="1000"
maxThreads="200"
maxHttpHeaderSize="16384"
address="127.0.0.1"
redirectPort="8443"/>
According to the catalina log the Broken pipes happend when trying to write
the response :
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:128)
at
org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at
org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:174)
at
org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:163)
at
org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:242)
at
org.apache.coyote.http11.InternalNioOutputBuffer.endRequest(InternalNioOutputBuffer.java:121)
at
org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:761)
at org.apache.coyote.Response.action(Response.java:174)
at org.apache.coyote.Response.finish(Response.java:274)
at
org.apache.catalina.connector.OutputBuffer.close(OutputBuffer.java:319)
at
org.apache.catalina.connector.CoyoteWriter.close(CoyoteWriter.java:112)
at
networkComm.commands.HttpCommand.sendResponse(HttpCommand.java:224)
We do handle requests from mobile devices but this is not the biggest part
of the traffic, I don't have information on how many request it could
represent for the test we made.
Thanks,
Thomas
2015-03-27 14:08 GMT+01:00 Christopher Schultz <chris@christopherschultz.net
>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Thomas,
>
> On 3/26/15 12:16 PM, Thomas Boniface wrote:
> > Following the update of the tomcat 7 version from 7.0.28-4 to
> > 7.0.56-1 hoping to improve our application performances we
> > experienced a quick freeze of the application under a standard load
> > (that was handle by previous tomcat version without particular
> > issue).
> >
> > Our application is behind a nginx server and receives a high volume
> > of HTTP requests from client and is using servlet 3.0 async
> > features.
> >
> > While investigating we made following observations: 1/ catalina.out
> > shows a high number of IOException Broken pipe. 2/ a high number of
> > CLOSE_WAIT appear between nginx and tomcat.
> >
> > I have uploaded a catalina log, an lsof and a netstat capture made
> > while the issue was happening:
> > http://s000.tinyupload.com/index.php?file_id=85064286189504330545
> > (email size limit prevented me from attaching them to the
> > message). Any pointer is welcomed. Also is there a way to have more
> > details on connection status of tomcat ?
>
> What <Connector> are you using?
>
> Can you take a thread dump to see if threads are stuck somewhere?
>
> For the broken pipes... are they failing during reads or writes? Are
> you handling requests from a large number of limited- or
> unreliable-bandwidth clients -- like mobile devices?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
>
> iQIcBAEBCAAGBQJVFVZJAAoJEBzwKT+lPKRYwF0P/jM9+o7H0GEs4SiT8WGlVLNb
> lPF9lMmbGdfZd0y/4Chm7h0L693OqyttGpeXlDZgvC66YyEPV6rrR9pRvFN/F9dM
> KwJd29+5X7g2Mhm6A89CCYZOXvK9xeALt1diCnUth+QS65I97zgRcUD0L/kxdReZ
> g2BncEGdHGp/s7WkRpcI3YZ/dzOFiJQpEr8zUsqHjkGA5jLbwC6sYSa5dvOMKDFB
> 55vWJwLzdMxtvJPcATEvaXNmCGkQr6izGWewMDHstkcalafSZPa3DIUF4QJYwUzz
> YIaVaclJsffkkllRM+go739vnd4i7Y746pNtC3lrRwh27CtuIMcBPEP6uPNZ+Bc/
> aBmCaPBcGxevhALJ86Ha3TacPBiwIoKpMBimgzzSvjzPklKFFI/b865VZPLUd+Xh
> Y+FCW3NotmnDzkpW+u7qLbGvsLca1cTqwzJzucit3iBDA7YZzAneznEl/v8DetqX
> uKwuvm7CmLb1gHhxhFotA1EHjB1UIwHmT20eovwvBFWWlfjT6P0UUN/TXP1ju1Vi
> 1tPrX66ONheiyK8VXi2gj63RxmvlJKCrb2qG8/l4qcSNJ5OHzYijrtvAfGxeKWt8
> YahKqTuQCsfCHnh625SAqhGVav+nkL7cHV0ZGhUzX/hyrORmKFMK6GBhA+jGIbO3
> R52u4jQosMmfP5fSxl2S
> =IRiW
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat 7 upgrade from 7.0.28-4 to 7.0.56-1 causes application
freeze
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Thomas,
On 3/26/15 12:16 PM, Thomas Boniface wrote:
> Following the update of the tomcat 7 version from 7.0.28-4 to
> 7.0.56-1 hoping to improve our application performances we
> experienced a quick freeze of the application under a standard load
> (that was handle by previous tomcat version without particular
> issue).
>
> Our application is behind a nginx server and receives a high volume
> of HTTP requests from client and is using servlet 3.0 async
> features.
>
> While investigating we made following observations: 1/ catalina.out
> shows a high number of IOException Broken pipe. 2/ a high number of
> CLOSE_WAIT appear between nginx and tomcat.
>
> I have uploaded a catalina log, an lsof and a netstat capture made
> while the issue was happening:
> http://s000.tinyupload.com/index.php?file_id=85064286189504330545
> (email size limit prevented me from attaching them to the
> message). Any pointer is welcomed. Also is there a way to have more
> details on connection status of tomcat ?
What <Connector> are you using?
Can you take a thread dump to see if threads are stuck somewhere?
For the broken pipes... are they failing during reads or writes? Are
you handling requests from a large number of limited- or
unreliable-bandwidth clients -- like mobile devices?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
iQIcBAEBCAAGBQJVFVZJAAoJEBzwKT+lPKRYwF0P/jM9+o7H0GEs4SiT8WGlVLNb
lPF9lMmbGdfZd0y/4Chm7h0L693OqyttGpeXlDZgvC66YyEPV6rrR9pRvFN/F9dM
KwJd29+5X7g2Mhm6A89CCYZOXvK9xeALt1diCnUth+QS65I97zgRcUD0L/kxdReZ
g2BncEGdHGp/s7WkRpcI3YZ/dzOFiJQpEr8zUsqHjkGA5jLbwC6sYSa5dvOMKDFB
55vWJwLzdMxtvJPcATEvaXNmCGkQr6izGWewMDHstkcalafSZPa3DIUF4QJYwUzz
YIaVaclJsffkkllRM+go739vnd4i7Y746pNtC3lrRwh27CtuIMcBPEP6uPNZ+Bc/
aBmCaPBcGxevhALJ86Ha3TacPBiwIoKpMBimgzzSvjzPklKFFI/b865VZPLUd+Xh
Y+FCW3NotmnDzkpW+u7qLbGvsLca1cTqwzJzucit3iBDA7YZzAneznEl/v8DetqX
uKwuvm7CmLb1gHhxhFotA1EHjB1UIwHmT20eovwvBFWWlfjT6P0UUN/TXP1ju1Vi
1tPrX66ONheiyK8VXi2gj63RxmvlJKCrb2qG8/l4qcSNJ5OHzYijrtvAfGxeKWt8
YahKqTuQCsfCHnh625SAqhGVav+nkL7cHV0ZGhUzX/hyrORmKFMK6GBhA+jGIbO3
R52u4jQosMmfP5fSxl2S
=IRiW
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org