You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Rallavagu <ra...@gmail.com> on 2014/05/03 00:22:01 UTC
BLOCKED threads
All,
Tomcat Version: 7.0.47
JVM Version: 1.7.0_51-b13
I see many blocked threads (90) in the thread dump. There are mainly two
monitors that block 69 threads.
One of them is below. It appears that it is simply trying to log.
----------------------------------------------------------------------------------
"http-bio-28080-exec-396" daemon prio=10 tid=0x00007fcbc814f000
nid=0x5804 runnable [0x00007fcc2144d000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.getStackTraceElement(Native Method)
at java.lang.Throwable.getOurStackTrace(Throwable.java:827)
- locked <0x00000007e1886340> (a java.util.NoSuchElementException)
at java.lang.Throwable.printStackTrace(Throwable.java:656)
- locked <0x00000007e207a5a8> (a java.io.PrintWriter)
at java.lang.Throwable.printStackTrace(Throwable.java:721)
at
java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157)
- locked <0x00000007008187e8> (a java.util.logging.SimpleFormatter)
at java.util.logging.StreamHandler.publish(StreamHandler.java:196)
- locked <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
at
java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
at java.util.logging.Logger.log(Logger.java:610)
at java.util.logging.Logger.doLog(Logger.java:631)
at java.util.logging.Logger.logp(Logger.java:831)
at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185)
at
org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
- locked <0x00000007e0ba5dd8> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
----------------------------------------------------------------------------------
The second one has the lock on StandardClassLoader.
----------------------------------------------------------------------------------
"http-bio-28080-exec-605" daemon prio=10 tid=0x00007fcbc82b8800
nid=0x77e6 runnable [0x00007fcb919d6000]
java.lang.Thread.State: RUNNABLE
at java.lang.ClassLoader.findLoadedClass0(Native Method)
at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1093)
at java.lang.ClassLoader.loadClass(ClassLoader.java:407)
- locked <0x0000000700810fc8> (a
org.apache.catalina.loader.StandardClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at
java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566)
at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296)
at java.util.ResourceBundle.getBundle(ResourceBundle.java:721)
at oracle.net.ns.Message11.getMessage(Message11.java:62)
at oracle.net.ns.NetException.getMessage(NetException.java:222)
at oracle.net.ano.AnoComm.b(Unknown Source)
at oracle.net.ano.AnoComm.o(Unknown Source)
at oracle.net.ano.Ano.a(Unknown Source)
at oracle.net.ano.Ano.negotiation(Unknown Source)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:407)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292)
at
oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
at
oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
at
org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
at
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1185)
at
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at
org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141)
at
org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
at
org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
at
org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
at
org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
at
org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
at
org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1395)
at
org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:387)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:334)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at
com.sun.proxy.$Proxy236.getOauth2AccessTokenByTokenId(Unknown Source)
----------------------------------------------------------------------------------
I need help explain these two locks. I am wondering how would class
loader would block those many threads. One clue is that App Dynamics
plugin is running for monitoring. However, I do not find it's class in
the stack trace. Looking for some clues here. Thanks in advance.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: BLOCKED threads
Posted by Mark Thomas <ma...@apache.org>.
On 03/05/2014 02:37, Martin Gainty wrote:
> Managing Provider Tokens
> Like the provider, the consumer must be responsible for managing the
> OAuth tokens. The necessary interface for managing the consumer tokens
> is OAuthConsumerTokenServices which are only accessible via factory method. Assuming that the consumer can leverage an active HTTP session, the default HttpSessionBasedTokenServices and HttpSessionBasedTokenServicesFactory should be adequate.
>
> so it appears that Spring is looking for an implmentor for OAuthConsumerTokenServices either
> HttpSessionBasedTokenServices OR
> HttpSessionBasedTokenServicesFactory
>
>
>
> http://spring-security-oauth.codehaus.org/oauth1.html#Managing_Provider_Tokens
>
> Martin
This response is so far off the reservation I can't begin to figure out
how you got to OAuth from the OP's question.
As per http://markmail.org/message/mukjphyw7kganuqi and
http://markmail.org/message/7x2tjic2azwfbw5o an unsubscription will follow.
Mark
>
>
>> Date: Fri, 2 May 2014 15:22:01 -0700
>> From: rallavagu@gmail.com
>> To: users@tomcat.apache.org
>> Subject: BLOCKED threads
>>
>> All,
>>
>> Tomcat Version: 7.0.47
>> JVM Version: 1.7.0_51-b13
>>
>> I see many blocked threads (90) in the thread dump. There are mainly two
>> monitors that block 69 threads.
>>
>> One of them is below. It appears that it is simply trying to log.
>> ----------------------------------------------------------------------------------
>> "http-bio-28080-exec-396" daemon prio=10 tid=0x00007fcbc814f000
>> nid=0x5804 runnable [0x00007fcc2144d000]
>> java.lang.Thread.State: RUNNABLE
>> at java.lang.Throwable.getStackTraceElement(Native Method)
>> at java.lang.Throwable.getOurStackTrace(Throwable.java:827)
>> - locked <0x00000007e1886340> (a java.util.NoSuchElementException)
>> at java.lang.Throwable.printStackTrace(Throwable.java:656)
>> - locked <0x00000007e207a5a8> (a java.io.PrintWriter)
>> at java.lang.Throwable.printStackTrace(Throwable.java:721)
>> at
>> java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157)
>> - locked <0x00000007008187e8> (a java.util.logging.SimpleFormatter)
>> at java.util.logging.StreamHandler.publish(StreamHandler.java:196)
>> - locked <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
>> at
>> java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
>> at java.util.logging.Logger.log(Logger.java:610)
>> at java.util.logging.Logger.doLog(Logger.java:631)
>> at java.util.logging.Logger.logp(Logger.java:831)
>> at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185)
>> at
>> org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>> at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
>> at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>> at
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
>> at
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
>> at
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>> - locked <0x00000007e0ba5dd8> (a
>> org.apache.tomcat.util.net.SocketWrapper)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>>
>> ----------------------------------------------------------------------------------
>>
>> The second one has the lock on StandardClassLoader.
>>
>> ----------------------------------------------------------------------------------
>>
>> "http-bio-28080-exec-605" daemon prio=10 tid=0x00007fcbc82b8800
>> nid=0x77e6 runnable [0x00007fcb919d6000]
>> java.lang.Thread.State: RUNNABLE
>> at java.lang.ClassLoader.findLoadedClass0(Native Method)
>> at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1093)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:407)
>> - locked <0x0000000700810fc8> (a
>> org.apache.catalina.loader.StandardClassLoader)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at
>> java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566)
>> at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
>> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400)
>> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
>> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
>> at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296)
>> at java.util.ResourceBundle.getBundle(ResourceBundle.java:721)
>> at oracle.net.ns.Message11.getMessage(Message11.java:62)
>> at oracle.net.ns.NetException.getMessage(NetException.java:222)
>> at oracle.net.ano.AnoComm.b(Unknown Source)
>> at oracle.net.ano.AnoComm.o(Unknown Source)
>> at oracle.net.ano.Ano.a(Unknown Source)
>> at oracle.net.ano.Ano.negotiation(Unknown Source)
>> at oracle.net.ns.NSProtocol.connect(NSProtocol.java:407)
>> at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966)
>> at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292)
>> at
>> oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
>> at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
>> at
>> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
>> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
>> at
>> org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
>> at
>> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>> at
>> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1185)
>> at
>> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
>> at
>> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
>> at
>> org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141)
>> at
>> org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
>> at
>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
>> at
>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
>> at
>> org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
>> at
>> org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
>> at
>> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1395)
>> at
>> org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:387)
>> at
>> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
>> at
>> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:334)
>> at
>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>> at
>> com.sun.proxy.$Proxy236.getOauth2AccessTokenByTokenId(Unknown Source)
>> ----------------------------------------------------------------------------------
>>
>> I need help explain these two locks. I am wondering how would class
>> loader would block those many threads. One clue is that App Dynamics
>> plugin is running for monitoring. However, I do not find it's class in
>> the stack trace. Looking for some clues here. Thanks in advance.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: BLOCKED threads
Posted by Martin Gainty <mg...@hotmail.com>.
Managing Provider Tokens
Like the provider, the consumer must be responsible for managing the
OAuth tokens. The necessary interface for managing the consumer tokens
is OAuthConsumerTokenServices which are only accessible via factory method. Assuming that the consumer can leverage an active HTTP session, the default HttpSessionBasedTokenServices and HttpSessionBasedTokenServicesFactory should be adequate.
so it appears that Spring is looking for an implmentor for OAuthConsumerTokenServices either
HttpSessionBasedTokenServices OR
HttpSessionBasedTokenServicesFactory
http://spring-security-oauth.codehaus.org/oauth1.html#Managing_Provider_Tokens
Martin
> Date: Fri, 2 May 2014 15:22:01 -0700
> From: rallavagu@gmail.com
> To: users@tomcat.apache.org
> Subject: BLOCKED threads
>
> All,
>
> Tomcat Version: 7.0.47
> JVM Version: 1.7.0_51-b13
>
> I see many blocked threads (90) in the thread dump. There are mainly two
> monitors that block 69 threads.
>
> One of them is below. It appears that it is simply trying to log.
> ----------------------------------------------------------------------------------
> "http-bio-28080-exec-396" daemon prio=10 tid=0x00007fcbc814f000
> nid=0x5804 runnable [0x00007fcc2144d000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.Throwable.getStackTraceElement(Native Method)
> at java.lang.Throwable.getOurStackTrace(Throwable.java:827)
> - locked <0x00000007e1886340> (a java.util.NoSuchElementException)
> at java.lang.Throwable.printStackTrace(Throwable.java:656)
> - locked <0x00000007e207a5a8> (a java.io.PrintWriter)
> at java.lang.Throwable.printStackTrace(Throwable.java:721)
> at
> java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157)
> - locked <0x00000007008187e8> (a java.util.logging.SimpleFormatter)
> at java.util.logging.StreamHandler.publish(StreamHandler.java:196)
> - locked <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
> at
> java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
> at java.util.logging.Logger.log(Logger.java:610)
> at java.util.logging.Logger.doLog(Logger.java:631)
> at java.util.logging.Logger.logp(Logger.java:831)
> at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185)
> at
> org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
> - locked <0x00000007e0ba5dd8> (a
> org.apache.tomcat.util.net.SocketWrapper)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
>
> ----------------------------------------------------------------------------------
>
> The second one has the lock on StandardClassLoader.
>
> ----------------------------------------------------------------------------------
>
> "http-bio-28080-exec-605" daemon prio=10 tid=0x00007fcbc82b8800
> nid=0x77e6 runnable [0x00007fcb919d6000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.ClassLoader.findLoadedClass0(Native Method)
> at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1093)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:407)
> - locked <0x0000000700810fc8> (a
> org.apache.catalina.loader.StandardClassLoader)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566)
> at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400)
> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1354)
> at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296)
> at java.util.ResourceBundle.getBundle(ResourceBundle.java:721)
> at oracle.net.ns.Message11.getMessage(Message11.java:62)
> at oracle.net.ns.NetException.getMessage(NetException.java:222)
> at oracle.net.ano.AnoComm.b(Unknown Source)
> at oracle.net.ano.AnoComm.o(Unknown Source)
> at oracle.net.ano.Ano.a(Unknown Source)
> at oracle.net.ano.Ano.negotiation(Unknown Source)
> at oracle.net.ns.NSProtocol.connect(NSProtocol.java:407)
> at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966)
> at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292)
> at
> oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
> at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
> at
> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
> at
> org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
> at
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
> at
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1185)
> at
> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
> at
> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
> at
> org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141)
> at
> org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
> at
> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
> at
> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
> at
> org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
> at
> org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
> at
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1395)
> at
> org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:387)
> at
> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
> at
> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:334)
> at
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at
> com.sun.proxy.$Proxy236.getOauth2AccessTokenByTokenId(Unknown Source)
> ----------------------------------------------------------------------------------
>
> I need help explain these two locks. I am wondering how would class
> loader would block those many threads. One clue is that App Dynamics
> plugin is running for monitoring. However, I do not find it's class in
> the stack trace. Looking for some clues here. Thanks in advance.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: BLOCKED threads
Posted by Rallavagu <ra...@gmail.com>.
Logs are written to local disk. Don't have data on disk stats. Thanks.
> On May 5, 2014, at 7:19 AM, Daniel Mikusa <dm...@gopivotal.com> wrote:
>
>> On May 3, 2014, at 9:08 PM, Rallavagu <ra...@gmail.com> wrote:
>>
>> Here is the thread BLOCKED waiting on another lock.
>>
>> "http-bio-28080-exec-613" daemon prio=10 tid=0x00007fcbac0e0800 nid=0x7897 waiting for monitor entry [0x00007fcb915d3000]
>> java.lang.Thread.State: BLOCKED (on object monitor)
>> at java.util.logging.StreamHandler.publish(StreamHandler.java:191)
>> - waiting to lock <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
>> at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
>> at java.util.logging.Logger.log(Logger.java:610)
>> at java.util.logging.Logger.doLog(Logger.java:631)
>> at java.util.logging.Logger.logp(Logger.java:831)
>> at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185)
>> at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
>> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
>> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
>> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
>> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
>> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>> - locked <0x00000007e1b0f808> (a org.apache.tomcat.util.net.SocketWrapper)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>
> Looks like this is waiting to log something. Where are you writing your logs? Is it a local disk or network disk? Is the disk busy or otherwise slow?
>
> Dan
>
>>
>> Locked ownable synchronizers:
>> - <0x00000007e1b0f8d0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
>>
>>> On 5/2/14, 9:19 PM, Christopher Schultz wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA256
>>>
>>> Rallavagu,
>>>
>>>> On 5/2/14, 6:22 PM, Rallavagu wrote:
>>>> Tomcat Version: 7.0.47 JVM Version: 1.7.0_51-b13
>>>>
>>>> I see many blocked threads (90) in the thread dump. There are
>>>> mainly two monitors that block 69 threads.
>>>>
>>>> One of them is below. It appears that it is simply trying to log.
>>>> ----------------------------------------------------------------------------------
>>>>
>>>> "http-bio-28080-exec-396" daemon prio=10 tid=0x00007fcbc814f000
>>>> nid=0x5804 runnable [0x00007fcc2144d000] java.lang.Thread.State:
>>>> RUNNABLE at java.lang.Throwable.getStackTraceElement(Native
>>>> Method)
>>>
>>> This thread is not blocked. What makes you think it is?
>>>
>>>> at java.lang.Throwable.getOurStackTrace(Throwable.java:827) -
>>>> locked <0x00000007e1886340> (a java.util.NoSuchElementException) at
>>>> java.lang.Throwable.printStackTrace(Throwable.java:656) - locked
>>>> <0x00000007e207a5a8> (a java.io.PrintWriter) at
>>>> java.lang.Throwable.printStackTrace(Throwable.java:721) at
>>>> java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157)
>>>> - locked <0x00000007008187e8> (a
>>>> java.util.logging.SimpleFormatter) at
>>>> java.util.logging.StreamHandler.publish(StreamHandler.java:196) -
>>>> locked <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
>>>> at
>>>> java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
>>>> at java.util.logging.Logger.log(Logger.java:610) at
>>>> java.util.logging.Logger.doLog(Logger.java:631) at
>>>> java.util.logging.Logger.logp(Logger.java:831) at
>>>> org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at
>>>> org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
>>>> at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
>>>>
>>>> at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>>
>>>> at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>>>>
>>>> at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>>
>>>> at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
>>>>
>>>> at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>>> at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>
>>>> at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>>> at
>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
>>>>
>>>> at
>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
>>>>
>>>> at
>>>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>>>>
>>>> - locked <0x00000007e0ba5dd8> (a
>>>> org.apache.tomcat.util.net.SocketWrapper) at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>
>>>> at java.lang.Thread.run(Thread.java:744)
>>>>
>>>> ----------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> The second one has the lock on StandardClassLoader.
>>>>
>>>> ----------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> "http-bio-28080-exec-605" daemon prio=10 tid=0x00007fcbc82b8800
>>>> nid=0x77e6 runnable [0x00007fcb919d6000] java.lang.Thread.State:
>>>> RUNNABLE at java.lang.ClassLoader.findLoadedClass0(Native Method)
>>>
>>> This thread is also not blocked.
>>>
>>>> at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1093) at
>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:407) - locked
>>>> <0x0000000700810fc8> (a
>>>> org.apache.catalina.loader.StandardClassLoader) at
>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
>>>> java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566)
>>> at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
>>>> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400) at
>>>> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
>>>> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
>>>> java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296) at
>>>> java.util.ResourceBundle.getBundle(ResourceBundle.java:721) at
>>>> oracle.net.ns.Message11.getMessage(Message11.java:62) at
>>>> oracle.net.ns.NetException.getMessage(NetException.java:222) at
>>>> oracle.net.ano.AnoComm.b(Unknown Source) at
>>>> oracle.net.ano.AnoComm.o(Unknown Source) at
>>>> oracle.net.ano.Ano.a(Unknown Source) at
>>>> oracle.net.ano.Ano.negotiation(Unknown Source) at
>>>> oracle.net.ns.NSProtocol.connect(NSProtocol.java:407) at
>>>> oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966) at
>>>> oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292) at
>>>> oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
>>> at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
>>>> at
>>>> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
>>>>
>>>> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1185)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
>>>>
>>>> at
>>>> org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141)
>>>>
>>>> at
>>>> org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
>>>>
>>>> at
>>>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
>>>>
>>>> at
>>>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
>>>>
>>>> at
>>>> org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
>>>>
>>>> at
>>>> org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
>>>>
>>>> at
>>>> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1395)
>>> at
>>>> org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:387)
>>>>
>>>> at
>>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
>>>>
>>>> at
>>>> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:334)
>>>>
>>>> at
>>>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
>>>>
>>>> at
>>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>>
>>>> at
>>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>>
>>>> at com.sun.proxy.$Proxy236.getOauth2AccessTokenByTokenId(Unknown
>>>> Source)
>>>> ----------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> I need help explain these two locks.
>>>
>>> Which two locks? The "locked" objects? Those are pretty standard
>>> object monitors. These two threads have no overlapping locks that I
>>> can see. Then again, it's pretty late at night for me...
>>>
>>>> I am wondering how would class loader would block those many
>>>> threads.
>>>
>>> Any class that modifies its internal state in a multi-threaded
>>> environment had better protect itself with such locks. If something is
>>> causing the class loading to take a long time, you might find threads
>>> waiting on each other.
>>>
>>> But I don't see any evidence that many (any?) threads are stuck at
>>> all. You showed two thread stack traces that appear to have little to
>>> do with one another.
>>>
>>>> One clue is that App Dynamics plugin is running for monitoring.
>>>> However, I do not find it's class in the stack trace. Looking for
>>>> some clues here. Thanks in advance.
>>>
>>> You need to provide more information. I don't see any problems, yet.
>>>
>>> - -chris
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1
>>> Comment: GPGTools - http://gpgtools.org
>>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>>
>>> iQIcBAEBCAAGBQJTZG5EAAoJEBzwKT+lPKRYuYsP/jIo4xoSHgPIi0E5fmmKgOW2
>>> AJkKh6X5rpeivPVRdAD3tf3QHWfJ3yX+YNSnfcgJlvHx2XsFfXgS+wxQViapg3IO
>>> MxEtSTlzmkDk1XwdWZr2yao7YRbDXJTtw5P8zi9Z8MSd3s0AZllvtTPTJP9WAKTt
>>> DgcaUZHSk6RIwrNZnrQmwYl4XpynWJSwC1nS/vp8PvVCn9ZizfcElN6IjvTNZ/BX
>>> jhstDVrVZyvP6+i9GAupTEnZRDzIl+q3czke8g6oBftw63URX74OczgKzCAjcfuz
>>> M9zCk0PTsj3hDxqiSqo9XmWlretYwy9NgWsW6+9DMI4PMIF8nIES7D0dMWsOvYCI
>>> ySMx7G7aPMxGrFxGNETJj6uvtOzfrytowJJfvUsA9MbxpIJlW7lcWtG9+maDYImO
>>> ikZ4E+dp/eKvlMmm7O6/0VvK0oXfzy1Tl36b5d34hbRXUw4flqLKrIxJ0bm8k1Qy
>>> XPIrzI8ZmMrSvIyU3bF6yzg0pO1Zk07knFroZdjhL5k2nKQozCCjPrvjOhiR8WQr
>>> 2TxkrbrxFCE8BygG80yysd2WG2AelXfBzRTs4DrKWv5LaDbCTtvK7nK5fOTqJLI+
>>> p/OJyZy8ie/yhEXp5oPASQ8mbYvxlImphzcMcmsmaErbuU/oBx3Au3zZU/7KQURT
>>> F51ctX8ZcTJBNlrPuUiC
>>> =COyv
>>> -----END PGP SIGNATURE-----
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: BLOCKED threads
Posted by Daniel Mikusa <dm...@gopivotal.com>.
On May 3, 2014, at 9:08 PM, Rallavagu <ra...@gmail.com> wrote:
> Here is the thread BLOCKED waiting on another lock.
>
> "http-bio-28080-exec-613" daemon prio=10 tid=0x00007fcbac0e0800 nid=0x7897 waiting for monitor entry [0x00007fcb915d3000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at java.util.logging.StreamHandler.publish(StreamHandler.java:191)
> - waiting to lock <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
> at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
> at java.util.logging.Logger.log(Logger.java:610)
> at java.util.logging.Logger.doLog(Logger.java:631)
> at java.util.logging.Logger.logp(Logger.java:831)
> at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185)
> at org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
> - locked <0x00000007e1b0f808> (a org.apache.tomcat.util.net.SocketWrapper)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
Looks like this is waiting to log something. Where are you writing your logs? Is it a local disk or network disk? Is the disk busy or otherwise slow?
Dan
>
> Locked ownable synchronizers:
> - <0x00000007e1b0f8d0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
>
> On 5/2/14, 9:19 PM, Christopher Schultz wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Rallavagu,
>>
>> On 5/2/14, 6:22 PM, Rallavagu wrote:
>>> Tomcat Version: 7.0.47 JVM Version: 1.7.0_51-b13
>>>
>>> I see many blocked threads (90) in the thread dump. There are
>>> mainly two monitors that block 69 threads.
>>>
>>> One of them is below. It appears that it is simply trying to log.
>>> ----------------------------------------------------------------------------------
>>>
>>> "http-bio-28080-exec-396" daemon prio=10 tid=0x00007fcbc814f000
>>> nid=0x5804 runnable [0x00007fcc2144d000] java.lang.Thread.State:
>>> RUNNABLE at java.lang.Throwable.getStackTraceElement(Native
>>> Method)
>>
>> This thread is not blocked. What makes you think it is?
>>
>>> at java.lang.Throwable.getOurStackTrace(Throwable.java:827) -
>>> locked <0x00000007e1886340> (a java.util.NoSuchElementException) at
>>> java.lang.Throwable.printStackTrace(Throwable.java:656) - locked
>>> <0x00000007e207a5a8> (a java.io.PrintWriter) at
>>> java.lang.Throwable.printStackTrace(Throwable.java:721) at
>>> java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157)
>>> - locked <0x00000007008187e8> (a
>>> java.util.logging.SimpleFormatter) at
>>> java.util.logging.StreamHandler.publish(StreamHandler.java:196) -
>>> locked <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
>>> at
>>> java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
>>> at java.util.logging.Logger.log(Logger.java:610) at
>>> java.util.logging.Logger.doLog(Logger.java:631) at
>>> java.util.logging.Logger.logp(Logger.java:831) at
>>> org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at
>>> org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
>>>
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>
>>> at
>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>>>
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
>>>
>>> at
>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>>>
>>>
>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>>>
>>>
>> at
>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
>>>
>>> at
>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
>>>
>>> at
>>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>>>
>>> - locked <0x00000007e0ba5dd8> (a
>>> org.apache.tomcat.util.net.SocketWrapper) at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>
>>> at java.lang.Thread.run(Thread.java:744)
>>>
>>> ----------------------------------------------------------------------------------
>>>
>>>
>>>
>>> The second one has the lock on StandardClassLoader.
>>>
>>> ----------------------------------------------------------------------------------
>>>
>>>
>>>
>>> "http-bio-28080-exec-605" daemon prio=10 tid=0x00007fcbc82b8800
>>> nid=0x77e6 runnable [0x00007fcb919d6000] java.lang.Thread.State:
>>> RUNNABLE at java.lang.ClassLoader.findLoadedClass0(Native Method)
>>
>> This thread is also not blocked.
>>
>>> at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1093) at
>>> java.lang.ClassLoader.loadClass(ClassLoader.java:407) - locked
>>> <0x0000000700810fc8> (a
>>> org.apache.catalina.loader.StandardClassLoader) at
>>> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
>>> java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566)
>>>
>>>
>> at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
>>> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400) at
>>> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
>>> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
>>> java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296) at
>>> java.util.ResourceBundle.getBundle(ResourceBundle.java:721) at
>>> oracle.net.ns.Message11.getMessage(Message11.java:62) at
>>> oracle.net.ns.NetException.getMessage(NetException.java:222) at
>>> oracle.net.ano.AnoComm.b(Unknown Source) at
>>> oracle.net.ano.AnoComm.o(Unknown Source) at
>>> oracle.net.ano.Ano.a(Unknown Source) at
>>> oracle.net.ano.Ano.negotiation(Unknown Source) at
>>> oracle.net.ns.NSProtocol.connect(NSProtocol.java:407) at
>>> oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966) at
>>> oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292) at
>>> oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
>>>
>>>
>> at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
>>> at
>>> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
>>>
>>> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
>>> at
>>> org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
>>>
>>> at
>>> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>>>
>>> at
>>> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1185)
>>>
>>> at
>>> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
>>>
>>> at
>>> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
>>>
>>> at
>>> org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141)
>>>
>>> at
>>> org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
>>>
>>> at
>>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
>>>
>>> at
>>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
>>>
>>> at
>>> org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
>>>
>>> at
>>> org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
>>>
>>> at
>>> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1395)
>>>
>>>
>> at
>>> org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:387)
>>>
>>> at
>>> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
>>>
>>> at
>>> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:334)
>>>
>>> at
>>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
>>>
>>> at
>>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>>
>>> at
>>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>>
>>> at com.sun.proxy.$Proxy236.getOauth2AccessTokenByTokenId(Unknown
>>> Source)
>>> ----------------------------------------------------------------------------------
>>>
>>>
>>>
>>> I need help explain these two locks.
>>
>> Which two locks? The "locked" objects? Those are pretty standard
>> object monitors. These two threads have no overlapping locks that I
>> can see. Then again, it's pretty late at night for me...
>>
>>> I am wondering how would class loader would block those many
>>> threads.
>>
>> Any class that modifies its internal state in a multi-threaded
>> environment had better protect itself with such locks. If something is
>> causing the class loading to take a long time, you might find threads
>> waiting on each other.
>>
>> But I don't see any evidence that many (any?) threads are stuck at
>> all. You showed two thread stack traces that appear to have little to
>> do with one another.
>>
>>> One clue is that App Dynamics plugin is running for monitoring.
>>> However, I do not find it's class in the stack trace. Looking for
>>> some clues here. Thanks in advance.
>>
>> You need to provide more information. I don't see any problems, yet.
>>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1
>> Comment: GPGTools - http://gpgtools.org
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iQIcBAEBCAAGBQJTZG5EAAoJEBzwKT+lPKRYuYsP/jIo4xoSHgPIi0E5fmmKgOW2
>> AJkKh6X5rpeivPVRdAD3tf3QHWfJ3yX+YNSnfcgJlvHx2XsFfXgS+wxQViapg3IO
>> MxEtSTlzmkDk1XwdWZr2yao7YRbDXJTtw5P8zi9Z8MSd3s0AZllvtTPTJP9WAKTt
>> DgcaUZHSk6RIwrNZnrQmwYl4XpynWJSwC1nS/vp8PvVCn9ZizfcElN6IjvTNZ/BX
>> jhstDVrVZyvP6+i9GAupTEnZRDzIl+q3czke8g6oBftw63URX74OczgKzCAjcfuz
>> M9zCk0PTsj3hDxqiSqo9XmWlretYwy9NgWsW6+9DMI4PMIF8nIES7D0dMWsOvYCI
>> ySMx7G7aPMxGrFxGNETJj6uvtOzfrytowJJfvUsA9MbxpIJlW7lcWtG9+maDYImO
>> ikZ4E+dp/eKvlMmm7O6/0VvK0oXfzy1Tl36b5d34hbRXUw4flqLKrIxJ0bm8k1Qy
>> XPIrzI8ZmMrSvIyU3bF6yzg0pO1Zk07knFroZdjhL5k2nKQozCCjPrvjOhiR8WQr
>> 2TxkrbrxFCE8BygG80yysd2WG2AelXfBzRTs4DrKWv5LaDbCTtvK7nK5fOTqJLI+
>> p/OJyZy8ie/yhEXp5oPASQ8mbYvxlImphzcMcmsmaErbuU/oBx3Au3zZU/7KQURT
>> F51ctX8ZcTJBNlrPuUiC
>> =COyv
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: BLOCKED threads
Posted by Rallavagu <ra...@gmail.com>.
Here is the thread BLOCKED waiting on another lock.
"http-bio-28080-exec-613" daemon prio=10 tid=0x00007fcbac0e0800
nid=0x7897 waiting for monitor entry [0x00007fcb915d3000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.logging.StreamHandler.publish(StreamHandler.java:191)
- waiting to lock <0x00000007008187b0> (a
java.util.logging.ConsoleHandler)
at
java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
at java.util.logging.Logger.log(Logger.java:610)
at java.util.logging.Logger.doLog(Logger.java:631)
at java.util.logging.Logger.logp(Logger.java:831)
at org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185)
at
org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
- locked <0x00000007e1b0f808> (a
org.apache.tomcat.util.net.SocketWrapper)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Locked ownable synchronizers:
- <0x00000007e1b0f8d0> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
On 5/2/14, 9:19 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Rallavagu,
>
> On 5/2/14, 6:22 PM, Rallavagu wrote:
>> Tomcat Version: 7.0.47 JVM Version: 1.7.0_51-b13
>>
>> I see many blocked threads (90) in the thread dump. There are
>> mainly two monitors that block 69 threads.
>>
>> One of them is below. It appears that it is simply trying to log.
>> ----------------------------------------------------------------------------------
>>
>> "http-bio-28080-exec-396" daemon prio=10 tid=0x00007fcbc814f000
>> nid=0x5804 runnable [0x00007fcc2144d000] java.lang.Thread.State:
>> RUNNABLE at java.lang.Throwable.getStackTraceElement(Native
>> Method)
>
> This thread is not blocked. What makes you think it is?
>
>> at java.lang.Throwable.getOurStackTrace(Throwable.java:827) -
>> locked <0x00000007e1886340> (a java.util.NoSuchElementException) at
>> java.lang.Throwable.printStackTrace(Throwable.java:656) - locked
>> <0x00000007e207a5a8> (a java.io.PrintWriter) at
>> java.lang.Throwable.printStackTrace(Throwable.java:721) at
>> java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157)
>> - locked <0x00000007008187e8> (a
>> java.util.logging.SimpleFormatter) at
>> java.util.logging.StreamHandler.publish(StreamHandler.java:196) -
>> locked <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
>> at
>> java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
>> at java.util.logging.Logger.log(Logger.java:610) at
>> java.util.logging.Logger.doLog(Logger.java:631) at
>> java.util.logging.Logger.logp(Logger.java:831) at
>> org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at
>> org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
>>
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>
>> at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>>
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
>>
>> at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>>
>>
> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>>
>>
> at
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
>>
>> at
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
>>
>> at
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>>
>> - locked <0x00000007e0ba5dd8> (a
>> org.apache.tomcat.util.net.SocketWrapper) at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:744)
>>
>> ----------------------------------------------------------------------------------
>>
>>
>>
>> The second one has the lock on StandardClassLoader.
>>
>> ----------------------------------------------------------------------------------
>>
>>
>>
>> "http-bio-28080-exec-605" daemon prio=10 tid=0x00007fcbc82b8800
>> nid=0x77e6 runnable [0x00007fcb919d6000] java.lang.Thread.State:
>> RUNNABLE at java.lang.ClassLoader.findLoadedClass0(Native Method)
>
> This thread is also not blocked.
>
>> at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1093) at
>> java.lang.ClassLoader.loadClass(ClassLoader.java:407) - locked
>> <0x0000000700810fc8> (a
>> org.apache.catalina.loader.StandardClassLoader) at
>> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
>> java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566)
>>
>>
> at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
>> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400) at
>> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
>> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
>> java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296) at
>> java.util.ResourceBundle.getBundle(ResourceBundle.java:721) at
>> oracle.net.ns.Message11.getMessage(Message11.java:62) at
>> oracle.net.ns.NetException.getMessage(NetException.java:222) at
>> oracle.net.ano.AnoComm.b(Unknown Source) at
>> oracle.net.ano.AnoComm.o(Unknown Source) at
>> oracle.net.ano.Ano.a(Unknown Source) at
>> oracle.net.ano.Ano.negotiation(Unknown Source) at
>> oracle.net.ns.NSProtocol.connect(NSProtocol.java:407) at
>> oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966) at
>> oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292) at
>> oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
>>
>>
> at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
>> at
>> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
>>
>> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
>> at
>> org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
>>
>> at
>> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>>
>> at
>> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1185)
>>
>> at
>> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
>>
>> at
>> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
>>
>> at
>> org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141)
>>
>> at
>> org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
>>
>> at
>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
>>
>> at
>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
>>
>> at
>> org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
>>
>> at
>> org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
>>
>> at
>> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1395)
>>
>>
> at
>> org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:387)
>>
>> at
>> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
>>
>> at
>> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:334)
>>
>> at
>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>
>> at com.sun.proxy.$Proxy236.getOauth2AccessTokenByTokenId(Unknown
>> Source)
>> ----------------------------------------------------------------------------------
>>
>>
>>
>> I need help explain these two locks.
>
> Which two locks? The "locked" objects? Those are pretty standard
> object monitors. These two threads have no overlapping locks that I
> can see. Then again, it's pretty late at night for me...
>
>> I am wondering how would class loader would block those many
>> threads.
>
> Any class that modifies its internal state in a multi-threaded
> environment had better protect itself with such locks. If something is
> causing the class loading to take a long time, you might find threads
> waiting on each other.
>
> But I don't see any evidence that many (any?) threads are stuck at
> all. You showed two thread stack traces that appear to have little to
> do with one another.
>
>> One clue is that App Dynamics plugin is running for monitoring.
>> However, I do not find it's class in the stack trace. Looking for
>> some clues here. Thanks in advance.
>
> You need to provide more information. I don't see any problems, yet.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJTZG5EAAoJEBzwKT+lPKRYuYsP/jIo4xoSHgPIi0E5fmmKgOW2
> AJkKh6X5rpeivPVRdAD3tf3QHWfJ3yX+YNSnfcgJlvHx2XsFfXgS+wxQViapg3IO
> MxEtSTlzmkDk1XwdWZr2yao7YRbDXJTtw5P8zi9Z8MSd3s0AZllvtTPTJP9WAKTt
> DgcaUZHSk6RIwrNZnrQmwYl4XpynWJSwC1nS/vp8PvVCn9ZizfcElN6IjvTNZ/BX
> jhstDVrVZyvP6+i9GAupTEnZRDzIl+q3czke8g6oBftw63URX74OczgKzCAjcfuz
> M9zCk0PTsj3hDxqiSqo9XmWlretYwy9NgWsW6+9DMI4PMIF8nIES7D0dMWsOvYCI
> ySMx7G7aPMxGrFxGNETJj6uvtOzfrytowJJfvUsA9MbxpIJlW7lcWtG9+maDYImO
> ikZ4E+dp/eKvlMmm7O6/0VvK0oXfzy1Tl36b5d34hbRXUw4flqLKrIxJ0bm8k1Qy
> XPIrzI8ZmMrSvIyU3bF6yzg0pO1Zk07knFroZdjhL5k2nKQozCCjPrvjOhiR8WQr
> 2TxkrbrxFCE8BygG80yysd2WG2AelXfBzRTs4DrKWv5LaDbCTtvK7nK5fOTqJLI+
> p/OJyZy8ie/yhEXp5oPASQ8mbYvxlImphzcMcmsmaErbuU/oBx3Au3zZU/7KQURT
> F51ctX8ZcTJBNlrPuUiC
> =COyv
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: BLOCKED threads
Posted by Rallavagu <ra...@gmail.com>.
Chris,
Thanks for the reply. Sorry for not being very clear. Actually, those
two threads holding the locks that caused other threads to be blocked.
Here is one of the threads that was blocked because of lock on
StandarClassLoader. There are around 47 threads in BLOCKED state waiting
on the same lock.
"http-bio-28080-exec-548" daemon prio=10 tid=0x00007fcbac060000
nid=0x76e0 waiting for monitor entry [0x00007fcc012ef000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
- waiting to lock <0x0000000700810fc8> (a
org.apache.catalina.loader.StandardClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at com.singularity.ee.agent.util.be.a(be.java:19)
at
com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.hb.f(hb.java:311)
at
com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.hb.c(hb.java:260)
at
com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.hb.b(hb.java:144)
at
com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.c.a(c.java:76)
at
com.singularity.ee.agent.appagent.services.transactionmonitor.common.e.a(e.java:488)
at
com.singularity.ee.agent.appagent.services.transactionmonitor.common.e.a(e.java:441)
at
com.singularity.ee.agent.appagent.services.transactionmonitor.common.e.a(e.java:375)
at
com.singularity.ee.agent.appagent.services.transactionmonitor.jdbc.c.a(c.java:32)
at
com.singularity.ee.agent.appagent.services.bciengine.a.onMethodEnd(a.java:62)
at
com.singularity.ee.agent.appagent.entrypoint.bciengine.FastMethodInterceptorDelegator.safeOnMethodEndNoReentrantCheck(FastMethodInterceptorDelegator.java:408)
at
com.singularity.ee.agent.appagent.entrypoint.bciengine.FastMethodInterceptorDelegator.safeOnMethodEnd(FastMethodInterceptorDelegator.java:350)
at
oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1491)
at
org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at
org.apache.tomcat.dbcp.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.hibernate.engine.jdbc.internal.proxy.AbstractStatementProxyHandler.continueInvocation(AbstractStatementProxyHandler.java:122)
at
org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81)
at com.sun.proxy.$Proxy397.executeQuery(Unknown Source)
at org.hibernate.loader.Loader.getResultSet(Loader.java:2031)
at
org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1832)
at
org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1811)
at org.hibernate.loader.Loader.doQuery(Loader.java:899)
at
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:341)
at org.hibernate.loader.Loader.doList(Loader.java:2516)
at org.hibernate.loader.Loader.doList(Loader.java:2502)
at
org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2332)
at org.hibernate.loader.Loader.list(Loader.java:2327)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:490)
at
org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:355)
at
org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1247)
at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101)
On 5/2/14, 9:19 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Rallavagu,
>
> On 5/2/14, 6:22 PM, Rallavagu wrote:
>> Tomcat Version: 7.0.47 JVM Version: 1.7.0_51-b13
>>
>> I see many blocked threads (90) in the thread dump. There are
>> mainly two monitors that block 69 threads.
>>
>> One of them is below. It appears that it is simply trying to log.
>> ----------------------------------------------------------------------------------
>>
>> "http-bio-28080-exec-396" daemon prio=10 tid=0x00007fcbc814f000
>> nid=0x5804 runnable [0x00007fcc2144d000] java.lang.Thread.State:
>> RUNNABLE at java.lang.Throwable.getStackTraceElement(Native
>> Method)
>
> This thread is not blocked. What makes you think it is?
>
>> at java.lang.Throwable.getOurStackTrace(Throwable.java:827) -
>> locked <0x00000007e1886340> (a java.util.NoSuchElementException) at
>> java.lang.Throwable.printStackTrace(Throwable.java:656) - locked
>> <0x00000007e207a5a8> (a java.io.PrintWriter) at
>> java.lang.Throwable.printStackTrace(Throwable.java:721) at
>> java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157)
>> - locked <0x00000007008187e8> (a
>> java.util.logging.SimpleFormatter) at
>> java.util.logging.StreamHandler.publish(StreamHandler.java:196) -
>> locked <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
>> at
>> java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
>> at java.util.logging.Logger.log(Logger.java:610) at
>> java.util.logging.Logger.doLog(Logger.java:631) at
>> java.util.logging.Logger.logp(Logger.java:831) at
>> org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at
>> org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
>> at
>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
>>
>> at
>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>
>> at
>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>>
>> at
>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>
>> at
>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
>>
>> at
>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>>
>>
> at
>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>
>> at
>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>>
>>
> at
>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
>>
>> at
>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
>>
>> at
>> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>>
>> - locked <0x00000007e0ba5dd8> (a
>> org.apache.tomcat.util.net.SocketWrapper) at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>
>> at java.lang.Thread.run(Thread.java:744)
>>
>> ----------------------------------------------------------------------------------
>>
>>
>>
>> The second one has the lock on StandardClassLoader.
>>
>> ----------------------------------------------------------------------------------
>>
>>
>>
>> "http-bio-28080-exec-605" daemon prio=10 tid=0x00007fcbc82b8800
>> nid=0x77e6 runnable [0x00007fcb919d6000] java.lang.Thread.State:
>> RUNNABLE at java.lang.ClassLoader.findLoadedClass0(Native Method)
>
> This thread is also not blocked.
>
>> at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1093) at
>> java.lang.ClassLoader.loadClass(ClassLoader.java:407) - locked
>> <0x0000000700810fc8> (a
>> org.apache.catalina.loader.StandardClassLoader) at
>> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
>> java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566)
>>
>>
> at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
>> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400) at
>> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
>> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
>> java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296) at
>> java.util.ResourceBundle.getBundle(ResourceBundle.java:721) at
>> oracle.net.ns.Message11.getMessage(Message11.java:62) at
>> oracle.net.ns.NetException.getMessage(NetException.java:222) at
>> oracle.net.ano.AnoComm.b(Unknown Source) at
>> oracle.net.ano.AnoComm.o(Unknown Source) at
>> oracle.net.ano.Ano.a(Unknown Source) at
>> oracle.net.ano.Ano.negotiation(Unknown Source) at
>> oracle.net.ns.NSProtocol.connect(NSProtocol.java:407) at
>> oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966) at
>> oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292) at
>> oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
>>
>>
> at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
>> at
>> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
>>
>> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
>> at
>> org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
>>
>> at
>> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>>
>> at
>> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1185)
>>
>> at
>> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
>>
>> at
>> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
>>
>> at
>> org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141)
>>
>> at
>> org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
>>
>> at
>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
>>
>> at
>> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
>>
>> at
>> org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
>>
>> at
>> org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
>>
>> at
>> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1395)
>>
>>
> at
>> org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:387)
>>
>> at
>> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
>>
>> at
>> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:334)
>>
>> at
>> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
>>
>> at
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>>
>> at
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>>
>> at com.sun.proxy.$Proxy236.getOauth2AccessTokenByTokenId(Unknown
>> Source)
>> ----------------------------------------------------------------------------------
>>
>>
>>
>> I need help explain these two locks.
>
> Which two locks? The "locked" objects? Those are pretty standard
> object monitors. These two threads have no overlapping locks that I
> can see. Then again, it's pretty late at night for me...
>
>> I am wondering how would class loader would block those many
>> threads.
>
> Any class that modifies its internal state in a multi-threaded
> environment had better protect itself with such locks. If something is
> causing the class loading to take a long time, you might find threads
> waiting on each other.
>
> But I don't see any evidence that many (any?) threads are stuck at
> all. You showed two thread stack traces that appear to have little to
> do with one another.
>
>> One clue is that App Dynamics plugin is running for monitoring.
>> However, I do not find it's class in the stack trace. Looking for
>> some clues here. Thanks in advance.
>
> You need to provide more information. I don't see any problems, yet.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJTZG5EAAoJEBzwKT+lPKRYuYsP/jIo4xoSHgPIi0E5fmmKgOW2
> AJkKh6X5rpeivPVRdAD3tf3QHWfJ3yX+YNSnfcgJlvHx2XsFfXgS+wxQViapg3IO
> MxEtSTlzmkDk1XwdWZr2yao7YRbDXJTtw5P8zi9Z8MSd3s0AZllvtTPTJP9WAKTt
> DgcaUZHSk6RIwrNZnrQmwYl4XpynWJSwC1nS/vp8PvVCn9ZizfcElN6IjvTNZ/BX
> jhstDVrVZyvP6+i9GAupTEnZRDzIl+q3czke8g6oBftw63URX74OczgKzCAjcfuz
> M9zCk0PTsj3hDxqiSqo9XmWlretYwy9NgWsW6+9DMI4PMIF8nIES7D0dMWsOvYCI
> ySMx7G7aPMxGrFxGNETJj6uvtOzfrytowJJfvUsA9MbxpIJlW7lcWtG9+maDYImO
> ikZ4E+dp/eKvlMmm7O6/0VvK0oXfzy1Tl36b5d34hbRXUw4flqLKrIxJ0bm8k1Qy
> XPIrzI8ZmMrSvIyU3bF6yzg0pO1Zk07knFroZdjhL5k2nKQozCCjPrvjOhiR8WQr
> 2TxkrbrxFCE8BygG80yysd2WG2AelXfBzRTs4DrKWv5LaDbCTtvK7nK5fOTqJLI+
> p/OJyZy8ie/yhEXp5oPASQ8mbYvxlImphzcMcmsmaErbuU/oBx3Au3zZU/7KQURT
> F51ctX8ZcTJBNlrPuUiC
> =COyv
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: BLOCKED threads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Rallavagu,
On 5/2/14, 6:22 PM, Rallavagu wrote:
> Tomcat Version: 7.0.47 JVM Version: 1.7.0_51-b13
>
> I see many blocked threads (90) in the thread dump. There are
> mainly two monitors that block 69 threads.
>
> One of them is below. It appears that it is simply trying to log.
> ----------------------------------------------------------------------------------
>
> "http-bio-28080-exec-396" daemon prio=10 tid=0x00007fcbc814f000
> nid=0x5804 runnable [0x00007fcc2144d000] java.lang.Thread.State:
> RUNNABLE at java.lang.Throwable.getStackTraceElement(Native
> Method)
This thread is not blocked. What makes you think it is?
> at java.lang.Throwable.getOurStackTrace(Throwable.java:827) -
> locked <0x00000007e1886340> (a java.util.NoSuchElementException) at
> java.lang.Throwable.printStackTrace(Throwable.java:656) - locked
> <0x00000007e207a5a8> (a java.io.PrintWriter) at
> java.lang.Throwable.printStackTrace(Throwable.java:721) at
> java.util.logging.SimpleFormatter.format(SimpleFormatter.java:157)
> - locked <0x00000007008187e8> (a
> java.util.logging.SimpleFormatter) at
> java.util.logging.StreamHandler.publish(StreamHandler.java:196) -
> locked <0x00000007008187b0> (a java.util.logging.ConsoleHandler)
> at
> java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:105)
> at java.util.logging.Logger.log(Logger.java:610) at
> java.util.logging.Logger.doLog(Logger.java:631) at
> java.util.logging.Logger.logp(Logger.java:831) at
> org.apache.juli.logging.DirectJDKLog.log(DirectJDKLog.java:185) at
> org.apache.juli.logging.DirectJDKLog.error(DirectJDKLog.java:151)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
>
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
>
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
>
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
>
>
at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
>
>
at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
>
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:603)
>
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
>
> - locked <0x00000007e0ba5dd8> (a
> org.apache.tomcat.util.net.SocketWrapper) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:744)
>
> ----------------------------------------------------------------------------------
>
>
>
> The second one has the lock on StandardClassLoader.
>
> ----------------------------------------------------------------------------------
>
>
>
> "http-bio-28080-exec-605" daemon prio=10 tid=0x00007fcbc82b8800
> nid=0x77e6 runnable [0x00007fcb919d6000] java.lang.Thread.State:
> RUNNABLE at java.lang.ClassLoader.findLoadedClass0(Native Method)
This thread is also not blocked.
> at java.lang.ClassLoader.findLoadedClass(ClassLoader.java:1093) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:407) - locked
> <0x0000000700810fc8> (a
> org.apache.catalina.loader.StandardClassLoader) at
> java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
> java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2566)
>
>
at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1436)
> at java.util.ResourceBundle.findBundle(ResourceBundle.java:1400) at
> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
> java.util.ResourceBundle.findBundle(ResourceBundle.java:1354) at
> java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1296) at
> java.util.ResourceBundle.getBundle(ResourceBundle.java:721) at
> oracle.net.ns.Message11.getMessage(Message11.java:62) at
> oracle.net.ns.NetException.getMessage(NetException.java:222) at
> oracle.net.ano.AnoComm.b(Unknown Source) at
> oracle.net.ano.AnoComm.o(Unknown Source) at
> oracle.net.ano.Ano.a(Unknown Source) at
> oracle.net.ano.Ano.negotiation(Unknown Source) at
> oracle.net.ns.NSProtocol.connect(NSProtocol.java:407) at
> oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966) at
> oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292) at
> oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
>
>
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
> at
> oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
>
> at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
> at
> org.apache.tomcat.dbcp.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38)
>
> at
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
>
> at
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1185)
>
> at
> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
>
> at
> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
>
> at
> org.hibernate.service.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:141)
>
> at
> org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:292)
>
> at
> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297)
>
> at
> org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169)
>
> at
> org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction.doBegin(JdbcTransaction.java:67)
>
> at
> org.hibernate.engine.transaction.spi.AbstractTransactionImpl.begin(AbstractTransactionImpl.java:160)
>
> at
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1395)
>
>
at
> org.springframework.orm.hibernate4.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:387)
>
> at
> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:371)
>
> at
> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:334)
>
> at
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:105)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>
> at com.sun.proxy.$Proxy236.getOauth2AccessTokenByTokenId(Unknown
> Source)
> ----------------------------------------------------------------------------------
>
>
>
> I need help explain these two locks.
Which two locks? The "locked" objects? Those are pretty standard
object monitors. These two threads have no overlapping locks that I
can see. Then again, it's pretty late at night for me...
> I am wondering how would class loader would block those many
> threads.
Any class that modifies its internal state in a multi-threaded
environment had better protect itself with such locks. If something is
causing the class loading to take a long time, you might find threads
waiting on each other.
But I don't see any evidence that many (any?) threads are stuck at
all. You showed two thread stack traces that appear to have little to
do with one another.
> One clue is that App Dynamics plugin is running for monitoring.
> However, I do not find it's class in the stack trace. Looking for
> some clues here. Thanks in advance.
You need to provide more information. I don't see any problems, yet.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTZG5EAAoJEBzwKT+lPKRYuYsP/jIo4xoSHgPIi0E5fmmKgOW2
AJkKh6X5rpeivPVRdAD3tf3QHWfJ3yX+YNSnfcgJlvHx2XsFfXgS+wxQViapg3IO
MxEtSTlzmkDk1XwdWZr2yao7YRbDXJTtw5P8zi9Z8MSd3s0AZllvtTPTJP9WAKTt
DgcaUZHSk6RIwrNZnrQmwYl4XpynWJSwC1nS/vp8PvVCn9ZizfcElN6IjvTNZ/BX
jhstDVrVZyvP6+i9GAupTEnZRDzIl+q3czke8g6oBftw63URX74OczgKzCAjcfuz
M9zCk0PTsj3hDxqiSqo9XmWlretYwy9NgWsW6+9DMI4PMIF8nIES7D0dMWsOvYCI
ySMx7G7aPMxGrFxGNETJj6uvtOzfrytowJJfvUsA9MbxpIJlW7lcWtG9+maDYImO
ikZ4E+dp/eKvlMmm7O6/0VvK0oXfzy1Tl36b5d34hbRXUw4flqLKrIxJ0bm8k1Qy
XPIrzI8ZmMrSvIyU3bF6yzg0pO1Zk07knFroZdjhL5k2nKQozCCjPrvjOhiR8WQr
2TxkrbrxFCE8BygG80yysd2WG2AelXfBzRTs4DrKWv5LaDbCTtvK7nK5fOTqJLI+
p/OJyZy8ie/yhEXp5oPASQ8mbYvxlImphzcMcmsmaErbuU/oBx3Au3zZU/7KQURT
F51ctX8ZcTJBNlrPuUiC
=COyv
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org