You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "B.V. Prasad" <hi...@yahoo.com> on 2012/03/31 03:12:26 UTC
High CPU with Tomcat 7.0.26 & 6.0.35
I am using Tomcat 6.0.16 in production. When I try to upgrade to Tomcat 6.0.35, I see high CPU (about 90% vs. about 20% for 6.0.16), all other things being equal, except that I had to set setBufferSize(16384) in HTTPResponse in order to disable chunked encoding (needed as some of my client appliances fail otherwise). I see the same high CPU issue even with Tomcat 7.0.26. Any help is greatly appreciated.
Thanks
Prasad
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU with Tomcat 7.0.26 & 6.0.35
Posted by "Marvin S. Addison" <ma...@gmail.com>.
> "http-443-205" daemon prio=10 tid=0x096a8400 nid=0x2a04 runnable [0x3089d000..0x3089e130]
> java.lang.Thread.State: RUNNABLE
> at org.apache.tomcat.jni.Socket.sendbb(Native Method)
> at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:711)
> at org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(InternalAprOutputBuffer.java:742)
> at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:127)
> at org.apache.coyote.http11.InternalAprOutputBuffer.doWrite(InternalAprOutputBuffer.java:552)
> at org.apache.coyote.Response.doWrite(Response.java:560)
We saw the same pattern as above in thread dumps from our hosts
exhibiting CPU spikes, but with a notable exception. In our case
whenever we captured traces, we saw at least _two_ threads in RUNNABLE
state at org.apache.tomcat.jni.Socket.sendbb(). There was also some
evidence that the number of threads in that state correlated with the
severity of the CPU spike.
M
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU with Tomcat 7.0.26 & 6.0.35
Posted by "B.V. Prasad" <hi...@yahoo.com>.
Th thread dump attached below is when the CPU was showing 55% (even after a day) even after all the incoming requests have been disabled. I tried with all latest: JDK 6.0.31, APR 1.4.6 & Tomcat native library 1.22. My -Xmx is 2048M, no changes to default PermGen size. I don't have OOME. The CPU is very high and erratic. Even tried incGc. I am using default settings for thread pool in server.xml (no changes made).
My server.xml has:
Connector port="80" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="443"
Connector port="443" protocol="HTTP/1.1" SSLEnabled="true" maxThreads="100"
acceptCount="100" scheme="https" secure="true" clientAuth="false" sslProtocol="TLS"
maxKeepAliveRequests="1" SSLCertificateFile="${catalina.home}/conf/xxx.cert"
SSLCertificateKeyFile="${catalina.home}/conf/xxx.key"
SSLCertificateChainFile="${catalina.home}/conf/ca.cert"
Connector port="8009" protocol="AJP/1.3" redirectPort="443"
I also set jarscanning to false.
2012-04-06 07:05:52
Full thread dump Java HotSpot(TM) Server VM (10.0-b23 mixed mode):
"http-80-6" daemon prio=10 tid=0x0a645c00 nid=0x3769 in Object.wait() [0x2fb0b000..0x2fb0be30]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x3ad68cd8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1631)
- locked <0x3ad68cd8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1656)
at java.lang.Thread.run(Thread.java:619)
"http-443-227" daemon prio=10 tid=0x0a11a000 nid=0x2a1a in Object.wait() [0x301a7000..0x301a8030]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x3a312c70> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1631)
- locked <0x3a312c70> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1656)
at java.lang.Thread.run(Thread.java:619)
<<< 237 more threads in the same state for both 80 & 443 together >>>
"http-443-205" daemon prio=10 tid=0x096a8400 nid=0x2a04 runnable [0x3089d000..0x3089e130]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.sendbb(Native Method)
at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:711)
at org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(InternalAprOutputBuffer.java:742)
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:127)
at org.apache.coyote.http11.InternalAprOutputBuffer.doWrite(InternalAprOutputBuffer.java:552)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:364)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:368)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:392)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:381)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:83)
at com.mycompany.xxx.xxx.process(xxx.java:222)
at com.mycompany.xxx.xxx.doPost(xxx.java:111)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:877)
at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:594)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1675)
at java.lang.Thread.run(Thread.java:619)
"ajp-8009-Acceptor-0" daemon prio=10 tid=0x333dfc00 nid=0xf54 runnable [0x0494a000..0x0494af30]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1243)
"ajp-8009-CometPoller-0" daemon prio=10 tid=0x334e7c00 nid=0xf53 in Object.wait() [0x048f9000..0x048f9db0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x384d4378> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1426)
- locked <0x384d4378> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
"ajp-8009-Poller-0" daemon prio=10 tid=0x33729000 nid=0xf52 in Object.wait() [0x048a8000..0x048a8e30]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x38484270> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1426)
- locked <0x38484270> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
"http-443-Acceptor-0" daemon prio=10 tid=0x33728800 nid=0xf4c runnable [0x04713000..0x04713f30]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1243)
"http-443-CometPoller-0" daemon prio=10 tid=0x337bac00 nid=0xf4b in Object.wait() [0x046c2000..0x046c2db0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x38482310> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1426)
- locked <0x38482310> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
"http-443-Poller-0" daemon prio=10 tid=0x337ba400 nid=0xf48 in Object.wait() [0x0721f000..0x07220130]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x38482178> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1426)
- locked <0x38482178> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
"http-80-Acceptor-0" daemon prio=10 tid=0x3372e400 nid=0xf47 runnable [0x04620000..0x04620fb0]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1243)
"http-80-Sendfile-0" daemon prio=10 tid=0x33531800 nid=0xf46 in Object.wait() [0x045cf000..0x045d0030]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x383905a8> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Sendfile.run(AprEndpoint.java:1900)
- locked <0x383905a8> (a org.apache.tomcat.util.net.AprEndpoint$Sendfile)
"http-80-CometPoller-0" daemon prio=10 tid=0x335fdc00 nid=0xf45 in Object.wait() [0x0457e000..0x0457eeb0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x38390408> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1426)
- locked <0x38390408> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
"http-80-Poller-0" daemon prio=10 tid=0x33535000 nid=0xf44 in Object.wait() [0x0452d000..0x0452df30]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x38390270> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1426)
- locked <0x38390270> (a org.apache.tomcat.util.net.AprEndpoint$Poller)
"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x33727c00 nid=0xf43 waiting on condition [0x044dc000..0x044dcdb0]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1631)
at java.lang.Thread.run(Thread.java:619)
"Thread-7" daemon prio=10 tid=0x33512400 nid=0xf42 waiting on condition [0x0448b000..0x0448be30]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95)
"Thread-6" daemon prio=10 tid=0x3378b800 nid=0xf41 waiting on condition [0x0443a000..0x0443b0b0]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95)
"Thread-4" daemon prio=10 tid=0x3378c800 nid=0xf3f waiting on condition [0x043e9000..0x043e9fb0]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95)
"Thread-3" daemon prio=10 tid=0x337f7c00 nid=0xf3d waiting on condition [0x00bbb000..0x00bbc030]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:95)
"GC Daemon" daemon prio=10 tid=0x3391cc00 nid=0xe8b in Object.wait() [0x00b6a000..0x00b6af30]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x37fcec68> (a sun.misc.GC$LatencyLock)
at sun.misc.GC$Daemon.run(GC.java:100)
- locked <0x37fcec68> (a sun.misc.GC$LatencyLock)
"Low Memory Detector" daemon prio=10 tid=0x08ece400 nid=0xdd7 runnable [0x00000000..0x00000000]
java.lang.Thread.State: RUNNABLE
"CompilerThread1" daemon prio=10 tid=0x08eccc00 nid=0xdd6 waiting on condition [0x00000000..0x00668578]
java.lang.Thread.State: RUNNABLE
"CompilerThread0" daemon prio=10 tid=0x08eca400 nid=0xdd5 waiting on condition [0x00000000..0x005e75f8]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x08ec9000 nid=0xdd4 runnable [0x00000000..0x00000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x08eae400 nid=0xdd3 in Object.wait() [0x0030c000..0x0030d030]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x37f1b410> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x37f1b410> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
"Reference Handler" daemon prio=10 tid=0x08ead400 nid=0xdd2 in Object.wait() [0x002bb000..0x002bbeb0]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x37f1b430> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x37f1b430> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x08e2a000 nid=0xdce runnable [0x00000000..0xbf7fd7a8]
java.lang.Thread.State: RUNNABLE
"VM Thread" prio=10 tid=0x08eaa000 nid=0xdd1 runnable
"GC task thread#0 (ParallelGC)" prio=10 tid=0x08e30c00 nid=0xdcf runnable
"GC task thread#1 (ParallelGC)" prio=10 tid=0x08e32000 nid=0xdd0 runnable
"VM Periodic Task Thread" prio=10 tid=0x08ecfc00 nid=0xdd8 waiting on condition
JNI global references: 811
Heap
PSYoungGen total 203456K, used 3336K [0xa9b90000, 0xb7f10000, 0xb7f10000)
eden space 172928K, 1% used [0xa9b90000,0xa9ed2270,0xb4470000)
from space 30528K, 0% used [0xb6140000,0xb6140000,0xb7f10000)
to space 29504K, 0% used [0xb4470000,0xb4470000,0xb6140000)
PSOldGen total 241088K, used 66235K [0x37f10000, 0x46a80000, 0xa9b90000)
object space 241088K, 27% used [0x37f10000,0x3bfbec28,0x46a80000)
PSPermGen total 23168K, used 22999K [0x33f10000, 0x355b0000, 0x37f10000)
object space 23168K, 99% used [0x33f10000,0x35585e78,0x355b0000)
________________________________
From: Mark Thomas <ma...@apache.org>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Saturday, March 31, 2012 4:30 AM
Subject: Re: High CPU with Tomcat 7.0.26 & 6.0.35
On 31/03/2012 02:12, B.V. Prasad wrote:
> I am using Tomcat 6.0.16 in production. When I try to upgrade to
> Tomcat 6.0.35, I see high CPU (about 90% vs. about 20% for 6.0.16),
> all other things being equal, except that I had to set
> setBufferSize(16384) in HTTPResponse in order to disable chunked
> encoding (needed as some of my client appliances fail otherwise). I
> see the same high CPU issue even with Tomcat 7.0.26. Any help is
> greatly appreciated.
Thread dumps.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: High CPU with Tomcat 7.0.26 & 6.0.35
Posted by Mark Thomas <ma...@apache.org>.
On 31/03/2012 02:12, B.V. Prasad wrote:
> I am using Tomcat 6.0.16 in production. When I try to upgrade to
> Tomcat 6.0.35, I see high CPU (about 90% vs. about 20% for 6.0.16),
> all other things being equal, except that I had to set
> setBufferSize(16384) in HTTPResponse in order to disable chunked
> encoding (needed as some of my client appliances fail otherwise). I
> see the same high CPU issue even with Tomcat 7.0.26. Any help is
> greatly appreciated.
Thread dumps.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org