You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Marvin Addison <ma...@gmail.com> on 2012/01/11 23:42:12 UTC
Excessive CPU w/APR Connectors on tomcat-native 1.1.22
We are seeing excessive CPU burn (top > 300% on multicore machine) in
multiple versions of Tomcat that use APR connectors exclusively. The
problem does not correlate with load. We initially saw it on 6.0.35
and subsequently on 7.0.23 as we attempted to upgrade around the
problem. We have determined that the component common to both
versions is tomcat-native 1.1.22. (We were not seeing this behavior
on our previous component mix of 6.0.26/1.1.20.)
In addition to the circumstantial evidence of version changes, we have
some JVM data that implicate tomcat-native. We've taken thread dumps
and YourKit snapshots (w/CPU sampling enabled) during the problem
periods and a consistent pattern appears: at least two connector
threads are in org.apache.tomcat.jni.Socket.sendbb(long, int, int)
throughout the period of CPU churn. The following instrumented thread
dump of active threads is illustrative of this pattern:
Stacks at 10:57:51 AM (uptime 5:13:38)
catalina-exec-1 [WAITING] CPU time: 2:03
sun.misc.Unsafe.park(boolean, long)
java.util.concurrent.locks.LockSupport.parkNanos(Object, long)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
java.util.concurrent.LinkedBlockingQueue.poll(long, TimeUnit)
org.apache.tomcat.util.threads.TaskQueue.poll(long, TimeUnit)
org.apache.tomcat.util.threads.TaskQueue.poll(long, TimeUnit)
java.util.concurrent.ThreadPoolExecutor.getTask()
java.util.concurrent.ThreadPoolExecutor$Worker.run()
java.lang.Thread.run()
catalina-exec-10 [RUNNABLE, IN_NATIVE] CPU time: 2:28
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()
org.apache.coyote.http11.InternalAprOutputBuffer.access$100(InternalAprOutputBuffer)
org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(ByteChunk,
Response)
org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(ByteChunk,
Response)
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(ByteChunk, Response)
org.apache.coyote.Response.doWrite(ByteChunk)
org.apache.catalina.connector.OutputBuffer.realWriteBytes(byte[], int, int)
org.apache.tomcat.util.buf.ByteChunk.append(byte[], int, int)
org.apache.catalina.connector.OutputBuffer.writeBytes(byte[], int, int)
org.apache.catalina.connector.OutputBuffer.write(byte[], int, int)
org.apache.catalina.connector.CoyoteOutputStream.write(byte[], int, int)
org.apache.catalina.servlets.DefaultServlet.copy(CacheEntry,
InputStream, ServletOutputStream)
org.apache.catalina.servlets.DefaultServlet.serveResource(HttpServletRequest,
HttpServletResponse, boolean)
org.apache.catalina.servlets.DefaultServlet.doGet(HttpServletRequest,
HttpServletResponse)
javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse)
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
edu.vt.middleware.servlet.filter.RequestDumperFilter.doFilter(ServletRequest,
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
com.github.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ServletRequest,
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
edu.vt.middleware.servlet.filter.CharacterEncodingFilter.doFilter(ServletRequest,
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Request, Response)
org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
org.apache.catalina.valves.AccessLogValve.invoke(Request, Response)
org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
org.apache.coyote.http11.AbstractHttp11Processor.process(SocketWrapper)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(SocketWrapper,
SocketStatus)
org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run()
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable)
java.util.concurrent.ThreadPoolExecutor$Worker.run()
java.lang.Thread.run()
catalina-exec-11 [RUNNABLE, IN_NATIVE] CPU time: 10:19
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()
org.apache.coyote.http11.InternalAprOutputBuffer.access$100(InternalAprOutputBuffer)
org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(ByteChunk,
Response)
org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(ByteChunk,
Response)
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(ByteChunk, Response)
org.apache.coyote.Response.doWrite(ByteChunk)
org.apache.catalina.connector.OutputBuffer.realWriteBytes(byte[], int, int)
org.apache.tomcat.util.buf.ByteChunk.append(byte[], int, int)
org.apache.catalina.connector.OutputBuffer.writeBytes(byte[], int, int)
org.apache.catalina.connector.OutputBuffer.write(byte[], int, int)
org.apache.catalina.connector.CoyoteOutputStream.write(byte[], int, int)
org.apache.catalina.servlets.DefaultServlet.copy(CacheEntry,
InputStream, ServletOutputStream)
org.apache.catalina.servlets.DefaultServlet.serveResource(HttpServletRequest,
HttpServletResponse, boolean)
org.apache.catalina.servlets.DefaultServlet.doGet(HttpServletRequest,
HttpServletResponse)
javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse)
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
edu.vt.middleware.servlet.filter.RequestDumperFilter.doFilter(ServletRequest,
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
com.github.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ServletRequest,
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
edu.vt.middleware.servlet.filter.CharacterEncodingFilter.doFilter(ServletRequest,
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Request, Response)
org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
org.apache.catalina.valves.AccessLogValve.invoke(Request, Response)
org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
org.apache.coyote.http11.AbstractHttp11Processor.process(SocketWrapper)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(SocketWrapper,
SocketStatus)
org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run()
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable)
java.util.concurrent.ThreadPoolExecutor$Worker.run()
java.lang.Thread.run()
catalina-exec-12 [WAITING] CPU time: 1:39
sun.misc.Unsafe.park(boolean, long)
java.util.concurrent.locks.LockSupport.parkNanos(Object, long)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
java.util.concurrent.LinkedBlockingQueue.poll(long, TimeUnit)
org.apache.tomcat.util.threads.TaskQueue.poll(long, TimeUnit)
org.apache.tomcat.util.threads.TaskQueue.poll(long, TimeUnit)
java.util.concurrent.ThreadPoolExecutor.getTask()
java.util.concurrent.ThreadPoolExecutor$Worker.run()
java.lang.Thread.run()
catalina-exec-3 [WAITING] CPU time: 2:26
sun.misc.Unsafe.park(boolean, long)
java.util.concurrent.locks.LockSupport.parkNanos(Object, long)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long)
java.util.concurrent.LinkedBlockingQueue.poll(long, TimeUnit)
org.apache.tomcat.util.threads.TaskQueue.poll(long, TimeUnit)
org.apache.tomcat.util.threads.TaskQueue.poll(long, TimeUnit)
java.util.concurrent.ThreadPoolExecutor.getTask()
java.util.concurrent.ThreadPoolExecutor$Worker.run()
java.lang.Thread.run()
catalina-exec-5 [RUNNABLE, IN_NATIVE] CPU time: 3:00
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()
org.apache.coyote.http11.InternalAprOutputBuffer.access$100(InternalAprOutputBuffer)
org.apache.coyote.http11.InternalAprOutputBuffer$SocketOutputBuffer.doWrite(ByteChunk,
Response)
org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(ByteChunk,
Response)
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(ByteChunk, Response)
org.apache.coyote.Response.doWrite(ByteChunk)
org.apache.catalina.connector.OutputBuffer.realWriteBytes(byte[], int, int)
org.apache.tomcat.util.buf.ByteChunk.flushBuffer()
org.apache.tomcat.util.buf.ByteChunk.append(byte[], int, int)
org.apache.catalina.connector.OutputBuffer.writeBytes(byte[], int, int)
org.apache.catalina.connector.OutputBuffer.write(byte[], int, int)
org.apache.catalina.connector.CoyoteOutputStream.write(byte[], int, int)
org.apache.catalina.servlets.DefaultServlet.copyRange(InputStream,
ServletOutputStream, long, long)
org.apache.catalina.servlets.DefaultServlet.copy(CacheEntry,
ServletOutputStream, DefaultServlet$Range)
org.apache.catalina.servlets.DefaultServlet.serveResource(HttpServletRequest,
HttpServletResponse, boolean)
org.apache.catalina.servlets.DefaultServlet.doGet(HttpServletRequest,
HttpServletResponse)
javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse)
javax.servlet.http.HttpServlet.service(ServletRequest, ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
edu.vt.middleware.servlet.filter.RequestDumperFilter.doFilter(ServletRequest,
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
com.github.inspektr.common.web.ClientInfoThreadLocalFilter.doFilter(ServletRequest,
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
edu.vt.middleware.servlet.filter.CharacterEncodingFilter.doFilter(ServletRequest,
ServletResponse, FilterChain)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ServletRequest,
ServletResponse)
org.apache.catalina.core.StandardWrapperValve.invoke(Request, Response)
org.apache.catalina.core.StandardContextValve.invoke(Request, Response)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(Request, Response)
org.apache.catalina.core.StandardHostValve.invoke(Request, Response)
org.apache.catalina.valves.ErrorReportValve.invoke(Request, Response)
org.apache.catalina.valves.AccessLogValve.invoke(Request, Response)
org.apache.catalina.core.StandardEngineValve.invoke(Request, Response)
org.apache.catalina.connector.CoyoteAdapter.service(Request, Response)
org.apache.coyote.http11.AbstractHttp11Processor.process(SocketWrapper)
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(SocketWrapper,
SocketStatus)
org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run()
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Runnable)
java.util.concurrent.ThreadPoolExecutor$Worker.run()
java.lang.Thread.run()
ContainerBackgroundProcessor[StandardEngine[Catalina]] [RUNNABLE,
IN_NATIVE] CPU time: 23:33
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery()
org.apache.catalina.session.JDBCStore.load(String)
org.apache.catalina.session.StoreBase.processExpires()
org.apache.catalina.session.PersistentManagerBase.processExpires()
org.apache.catalina.session.ManagerBase.backgroundProcess()
org.apache.catalina.core.ContainerBase.backgroundProcess()
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(Container,
ClassLoader)
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(Container,
ClassLoader)
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(Container,
ClassLoader)
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run()
java.lang.Thread.run()
pool-10-thread-1 [WAITING] CPU time: 4:11
sun.misc.Unsafe.park(boolean, long)
java.util.concurrent.locks.LockSupport.park(Object)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await()
java.util.concurrent.LinkedBlockingQueue.take()
java.util.concurrent.ThreadPoolExecutor.getTask()
java.util.concurrent.ThreadPoolExecutor$Worker.run()
java.lang.Thread.run()
Shown threads: 8 of 164
Graphs of CPU usage over time show a sharp increase when a second
thread enters sendbb; conversely there is a sharp decrease as soon as
all but a single thread drop out of the method. Additionally, there
may be a correlation with CPU usage and the number of threads in
sendbb; for example, the CPU burn may be greater when three threads
are in sendbb versus two.
This feels like a concurrency bug: hard to reproduce, sporadic, and
correlates with number of threads acting on the same code path.
Please let me know if you'd like me to do anything further that may
help determine whether this is, in fact, a bug. I'm happy to create a
bug report if needed.
M
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Excessive CPU w/APR Connectors on tomcat-native 1.1.22
Posted by Marvin Addison <ma...@gmail.com>.
Brief follow up on CPU spike issue. In an attempt to work around the
problem via configuration changes, we have swapped out APR connectors
with NIO using an equivalent configuration. (The only meaningful
changes are SSL configuration directives.) Since swapping out
connectors over the weekend, we have not had any CPU spikes. That's
long enough for me to consider it a suitable workaround.
M
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Excessive CPU w/APR Connectors on tomcat-native 1.1.22
Posted by Marvin Addison <ma...@gmail.com>.
> Can you confirm whether or not the issue exists with 6.0.26 and 1.1.22?
I cannot. We have tried repeatedly to reproduce this problem in a
test environment where such experimentation is tolerated, but the
problem simply does not manifest using available load testing tools.
We attempted to try 7.0.23+1.2.20 in production, but I couldn't get a
working configuration for the supported SSL/TLS protocols we require.
> How long do the periods of high CPU usage last?
Varies quite a bit. Based on data from our enterprise monitoring
system, the problem lasts as little as 5 min or less and has lasted as
much as 25 min. YourKit data from the incidents that have happened in
the past 24h shows similarly wide variation:
2m40s
1m38s
13m11s
8s
> How sure are you that the CPU is being burned in the threads that are in
> sendbb?
I'm certain of nothing, but I can share the evidence. YourKit CPU
profiling data shows that request processing threads consume the
majority of resources during these periods. No surprise there.
However, the only method that is consistently executed during these
periods is sendbb() by two or more threads. For the last problem
period, which happens to be the shortest (8s), catalina-exec threads
accounted for 89% of CPU use. Following are the RUNNABLE threads at
1s sample intervals during that period:
Stacks at 09:11:01 AM (uptime 1 day 3:26:47)
1. catalina-exec-45 [RUNNABLE, IN_NATIVE] CPU time: 1:57
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:32
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
Stacks at 09:11:02 AM (uptime 1 day 3:26:48)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:21
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:33
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
Stacks at 09:11:03 AM (uptime 1 day 3:26:49)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:22
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:34
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
Stacks at 09:11:04 AM (uptime 1 day 3:26:50)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:23
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:35
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
Stacks at 09:11:05 AM (uptime 1 day 3:26:51)
1. catalina-exec-38 [RUNNABLE, IN_NATIVE] CPU time: 1:28
java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int)
2. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:24
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
3. catalina-exec-44 [RUNNABLE] CPU time: 1:43
java.net.SocketInputStream.socketRead0(FileDescriptor, byte[], int, int, int)
4. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:35
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
Stacks at 09:11:06 AM (uptime 1 day 3:26:52)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:25
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:36
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
Stacks at 09:11:07 AM (uptime 1 day 3:26:53)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:25
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:37
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
3. catalina-exec-52 [RUNNABLE] CPU time: 0:04
java.lang.Throwable.fillInStackTrace()
Stacks at 09:11:08 AM (uptime 1 day 3:26:54)
1. catalina-exec-39 [RUNNABLE, IN_NATIVE] CPU time: 1:26
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
2. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:38
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
Stacks at 09:11:09 AM (uptime 1 day 3:26:55)
1. catalina-exec-51 [RUNNABLE, IN_NATIVE] CPU time: 13:39
org.apache.tomcat.jni.Socket.sendbb(long, int, int)
After the last sample, CPU usage drops precipitously. Our application
is a Web SSO product that writes relatively small response payloads.
It's hard to explain two request processing threads that take several
seconds to write small response payloads without citing a bug in the
servlet container.
M
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Excessive CPU w/APR Connectors on tomcat-native 1.1.22
Posted by Mark Thomas <ma...@apache.org>.
On 11/01/2012 22:42, Marvin Addison wrote:
> We are seeing excessive CPU burn (top > 300% on multicore machine) in
> multiple versions of Tomcat that use APR connectors exclusively. The
> problem does not correlate with load. We initially saw it on 6.0.35
> and subsequently on 7.0.23 as we attempted to upgrade around the
> problem. We have determined that the component common to both
> versions is tomcat-native 1.1.22. (We were not seeing this behavior
> on our previous component mix of 6.0.26/1.1.20.)
Can you confirm whether or not the issue exists with 6.0.26 and 1.1.22?
It would be helpful to try and track down which component is the root of
the issue.
How long do the periods of high CPU usage last?
> Graphs of CPU usage over time show a sharp increase when a second
> thread enters sendbb; conversely there is a sharp decrease as soon as
> all but a single thread drop out of the method. Additionally, there
> may be a correlation with CPU usage and the number of threads in
> sendbb; for example, the CPU burn may be greater when three threads
> are in sendbb versus two.
How sure are you that the CPU is being burned in the threads that are in
sendbb? Just because the CPU usage correlates with threads being in that
method it doesn't necessarily mean that is where the CPU is being used.
> This feels like a concurrency bug: hard to reproduce, sporadic, and
> correlates with number of threads acting on the same code path.
> Please let me know if you'd like me to do anything further that may
> help determine whether this is, in fact, a bug. I'm happy to create a
> bug report if needed.
Answers to the above questions would help with the analysis of this
issue. Assuming that this is a concurrency issue, then code inspection
is likely to be the best chance of finding the issue. If we can get to a
point where we can say upgrading this one component from x.y.z to
x.y.z+1 triggers the issue that will narrow down where we have to look.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org