You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "Eric B." <eb...@hotmail.com> on 2009/06/22 22:33:51 UTC
dbcp pool evictor deadlock?
Hi,
I appologize in advance for the cross-post, but I'm really not sure if this
is a problem with the dbcp code or my configuration.
I ran into the most troublesome deadlock over the weekend on my server, and
cannot figure out how/or why this deadlock occured. Via jconsole, I was
able to get thread information of the deadlock, but that hasn't helped me
diagnose the problem any further and/or determine if it is a config error,
or something more critical in the tomcat pool.
>From what I can tell, there is a race condition where the evictor was
triggered at the exact same moment as a resource was being added back into
the pool. The evictor was waiting for the resource to be added to the pool,
and the resource was waiting for the evictor to finish. However, I'm not
sure if I am correct or not.
My jndi settings are as follows:
<!-- Database resource -->
<Resource name="jdbc/db" auth="Container"
type="javax.sql.DataSource" username="db" password="db"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/myapp?autoReconnect=true"
maxActive="100" maxIdle="4"
validationQuery="select 1"
testOnBorrow="true"
testWhileIdle="true"
timeBetweenEvictionRunsMillis="10000"
minEvictableIdleTimeMillis="60000" />
My thread trace is the following:
THREAD 1:
Name: Timer-1
State: BLOCKED on org.apache.tomcat.dbcp.dbcp.PoolableConnection@1e667871
owned by: scheduling.QuartzInternal_Worker-0
Total blocked: 1 Total waited: 15,342
Stack trace:
org.apache.tomcat.dbcp.dbcp.AbandonedTrace.addTrace(AbandonedTrace.java:175)
org.apache.tomcat.dbcp.dbcp.AbandonedTrace.init(AbandonedTrace.java:92)
org.apache.tomcat.dbcp.dbcp.AbandonedTrace.<init>(AbandonedTrace.java:82)
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.<init>(DelegatingStatement.java:61)
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.createStatement(DelegatingConnection.java:224)
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:331)
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312)
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.evict(GenericObjectPool.java:1217)
- locked org.apache.tomcat.dbcp.pool.impl.GenericObjectPool@741a266
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1341)
java.util.TimerThread.mainLoop(Unknown Source)
java.util.TimerThread.run(Unknown Source)
THREAD 2:
Name: scheduling.QuartzInternal_Worker-0
State: BLOCKED on org.apache.tomcat.dbcp.pool.impl.GenericObjectPool@741a266
owned by: Timer-1
Total blocked: 156,031 Total waited: 206,465
Stack trace:
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.addObjectToPool(GenericObjectPool.java:1137)
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:1076)
org.apache.tomcat.dbcp.dbcp.PoolableConnection.close(PoolableConnection.java:87)
- locked org.apache.tomcat.dbcp.dbcp.PoolableConnection@1e667871
org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:96)
org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:451)
org.hibernate.jdbc.ConnectionManager.cleanup(ConnectionManager.java:385)
org.hibernate.jdbc.ConnectionManager.close(ConnectionManager.java:324)
org.hibernate.impl.SessionImpl.close(SessionImpl.java:298)
org.springframework.orm.hibernate3.SessionFactoryUtils.closeSession(SessionFactoryUtils.java:791)
org.springframework.orm.hibernate3.SessionFactoryUtils.closeSessionOrRegisterDeferredClose(SessionFactoryUtils.java:777)
org.springframework.orm.hibernate3.HibernateTransactionManager.doCleanupAfterCompletion(HibernateTransactionManager.java:733)
org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:989)
org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:782)
org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
$Proxy8.execute(Unknown Source)
sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
java.lang.reflect.Method.invoke(Unknown Source)
org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:276)
org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:260)
org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
org.quartz.core.JobRunShell.run(JobRunShell.java:203)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
Any help or insight into the problem and/or the correct dbcp settings would
be appreciated to ensure that this doesn't happen again.
Thanks,
Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by "Eric B." <eb...@hotmail.com>.
"Filip Hanik - Dev Lists" <de...@hanik.com> wrote in message
news:4A40EDDD.2000305@hanik.com...
> first, sorry for bringing this thread to dev, that should have never
> happened. I fat fingered the keyboard.
I cross posted my response to this, so the thread can continue on the user
list, if needed.
>>>> I think the issue is with that createConnection method of DBCP
>>>> synchronization. We have taken Thread dump to do above analysis.
>>>>
>>> Yep, known issues with commons-pool. Should be fixed in 1.5.1. Trunk has
>>> been updated. Proposed for 6.0.x and 5.5.x. Alternatively, there is the
>>> new JDBC pool module.
>>>
>>
>> I'm a little confused now. Filip pointed me to the jdbp-1.0.5 package.
>> Is that just repackaged versions of dbcp 1.2.2/pool 1.5? If not, what
>> versions of dbcp/pool are contained within there?
>>
> its a different package all together. Similar features, different
> implementation.
Is the implementation stable and fully tested? Or is it still undergoing
development?
Thanks,
Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
first, sorry for bringing this thread to dev, that should have never
happened. I fat fingered the keyboard.
Eric B. wrote:
> "Mark Thomas" <ma...@apache.org> wrote in message
> news:4A40E12C.2070504@apache.org...
>
>> Narendra Sarkar wrote:
>>
>>> Hi,
>>> We have a multi threaded environment. We have noticed that thread blocks
>>> for
>>> 10 to 20 minutes due DBCP getConnection method call. Then, we
>>> investigated
>>> the source code of DBCP(commons-dbcp-1.2.1-PII.jar) and noticed that
>>> createConnection method is synchronized. This problem occur more
>>> frequently
>>> when we have lower value of MaxIdleConnection parameter. After increasing
>>> the value of MaxIdleConnection parameter, frequency of blocking get
>>> reduced.
>>> We then replaced DBCP connection pooling with Oracle Connection pooling
>>> (ojdbc14.jar) and never encountered blocking issue.
>>>
>>> I think the issue is with that createConnection method of DBCP
>>> synchronization. We have taken Thread dump to do above analysis.
>>>
>> Yep, known issues with commons-pool. Should be fixed in 1.5.1. Trunk has
>> been updated. Proposed for 6.0.x and 5.5.x. Alternatively, there is the
>> new JDBC pool module.
>>
>
> I'm a little confused now. Filip pointed me to the jdbp-1.0.5 package. Is
> that just repackaged versions of dbcp 1.2.2/pool 1.5? If not, what versions
> of dbcp/pool are contained within there?
>
its a different package all together. Similar features, different
implementation.
> If the fix in commons-pool is in 1.5.1, am I still expecting concurrency
> problems if upgrading 1.5? What would be the best move?
>
if the issue is fixed in 1.5.1, you should upgrade to 1.5.1 if that's
what you want.
> Thanks,
>
> Eric
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by Mark Thomas <ma...@apache.org>.
Eric B. wrote:
> "Mark Thomas" <ma...@apache.org> wrote in message
> news:4A40E12C.2070504@apache.org...
>> Narendra Sarkar wrote:
>>> Hi,
>>> We have a multi threaded environment. We have noticed that thread blocks
>>> for
>>> 10 to 20 minutes due DBCP getConnection method call. Then, we
>>> investigated
>>> the source code of DBCP(commons-dbcp-1.2.1-PII.jar) and noticed that
>>> createConnection method is synchronized. This problem occur more
>>> frequently
>>> when we have lower value of MaxIdleConnection parameter. After increasing
>>> the value of MaxIdleConnection parameter, frequency of blocking get
>>> reduced.
>>> We then replaced DBCP connection pooling with Oracle Connection pooling
>>> (ojdbc14.jar) and never encountered blocking issue.
>>>
>>> I think the issue is with that createConnection method of DBCP
>>> synchronization. We have taken Thread dump to do above analysis.
>> Yep, known issues with commons-pool. Should be fixed in 1.5.1. Trunk has
>> been updated. Proposed for 6.0.x and 5.5.x. Alternatively, there is the
>> new JDBC pool module.
>
> I'm a little confused now. Filip pointed me to the jdbp-1.0.5 package. Is
> that just repackaged versions of dbcp 1.2.2/pool 1.5?
No, it is a completely new connection pool implementation.
> If not, what versions
> of dbcp/pool are contained within there?
>
> If the fix in commons-pool is in 1.5.1, am I still expecting concurrency
> problems if upgrading 1.5? What would be the best move?
pool 1.5 has a nasty bug. If you go the DBCP route you should use pool 1.5.1
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by "Eric B." <eb...@hotmail.com>.
"Mark Thomas" <ma...@apache.org> wrote in message
news:4A40E12C.2070504@apache.org...
> Narendra Sarkar wrote:
>> Hi,
>> We have a multi threaded environment. We have noticed that thread blocks
>> for
>> 10 to 20 minutes due DBCP getConnection method call. Then, we
>> investigated
>> the source code of DBCP(commons-dbcp-1.2.1-PII.jar) and noticed that
>> createConnection method is synchronized. This problem occur more
>> frequently
>> when we have lower value of MaxIdleConnection parameter. After increasing
>> the value of MaxIdleConnection parameter, frequency of blocking get
>> reduced.
>> We then replaced DBCP connection pooling with Oracle Connection pooling
>> (ojdbc14.jar) and never encountered blocking issue.
>>
>> I think the issue is with that createConnection method of DBCP
>> synchronization. We have taken Thread dump to do above analysis.
>
> Yep, known issues with commons-pool. Should be fixed in 1.5.1. Trunk has
> been updated. Proposed for 6.0.x and 5.5.x. Alternatively, there is the
> new JDBC pool module.
I'm a little confused now. Filip pointed me to the jdbp-1.0.5 package. Is
that just repackaged versions of dbcp 1.2.2/pool 1.5? If not, what versions
of dbcp/pool are contained within there?
If the fix in commons-pool is in 1.5.1, am I still expecting concurrency
problems if upgrading 1.5? What would be the best move?
Thanks,
Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by Mark Thomas <ma...@apache.org>.
Narendra Sarkar wrote:
> Hi,
> We have a multi threaded environment. We have noticed that thread blocks for
> 10 to 20 minutes due DBCP getConnection method call. Then, we investigated
> the source code of DBCP(commons-dbcp-1.2.1-PII.jar) and noticed that
> createConnection method is synchronized. This problem occur more frequently
> when we have lower value of MaxIdleConnection parameter. After increasing
> the value of MaxIdleConnection parameter, frequency of blocking get reduced.
> We then replaced DBCP connection pooling with Oracle Connection pooling
> (ojdbc14.jar) and never encountered blocking issue.
>
> I think the issue is with that createConnection method of DBCP
> synchronization. We have taken Thread dump to do above analysis.
Yep, known issues with commons-pool. Should be fixed in 1.5.1. Trunk has
been updated. Proposed for 6.0.x and 5.5.x. Alternatively, there is the
new JDBC pool module.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by Narendra Sarkar <na...@gmail.com>.
Hi,
We have a multi threaded environment. We have noticed that thread blocks for
10 to 20 minutes due DBCP getConnection method call. Then, we investigated
the source code of DBCP(commons-dbcp-1.2.1-PII.jar) and noticed that
createConnection method is synchronized. This problem occur more frequently
when we have lower value of MaxIdleConnection parameter. After increasing
the value of MaxIdleConnection parameter, frequency of blocking get reduced.
We then replaced DBCP connection pooling with Oracle Connection pooling
(ojdbc14.jar) and never encountered blocking issue.
I think the issue is with that createConnection method of DBCP
synchronization. We have taken Thread dump to do above analysis.
On Mon, Jun 22, 2009 at 4:33 PM, Eric B. <eb...@hotmail.com> wrote:
> Hi,
>
> I appologize in advance for the cross-post, but I'm really not sure if this
> is a problem with the dbcp code or my configuration.
>
> I ran into the most troublesome deadlock over the weekend on my server, and
> cannot figure out how/or why this deadlock occured. Via jconsole, I was
> able to get thread information of the deadlock, but that hasn't helped me
> diagnose the problem any further and/or determine if it is a config error,
> or something more critical in the tomcat pool.
>
> From what I can tell, there is a race condition where the evictor was
> triggered at the exact same moment as a resource was being added back into
> the pool. The evictor was waiting for the resource to be added to the
> pool,
> and the resource was waiting for the evictor to finish. However, I'm not
> sure if I am correct or not.
>
>
> My jndi settings are as follows:
> <!-- Database resource -->
> <Resource name="jdbc/db" auth="Container"
> type="javax.sql.DataSource" username="db" password="db"
> driverClassName="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/myapp?autoReconnect=true"
> maxActive="100" maxIdle="4"
> validationQuery="select 1"
> testOnBorrow="true"
> testWhileIdle="true"
> timeBetweenEvictionRunsMillis="10000"
> minEvictableIdleTimeMillis="60000" />
>
>
> My thread trace is the following:
> THREAD 1:
> Name: Timer-1
> State: BLOCKED on org.apache.tomcat.dbcp.dbcp.PoolableConnection@1e667871
> owned by: scheduling.QuartzInternal_Worker-0
> Total blocked: 1 Total waited: 15,342
>
> Stack trace:
>
> org.apache.tomcat.dbcp.dbcp.AbandonedTrace.addTrace(AbandonedTrace.java:175)
> org.apache.tomcat.dbcp.dbcp.AbandonedTrace.init(AbandonedTrace.java:92)
> org.apache.tomcat.dbcp.dbcp.AbandonedTrace.<init>(AbandonedTrace.java:82)
>
> org.apache.tomcat.dbcp.dbcp.DelegatingStatement.<init>(DelegatingStatement.java:61)
>
> org.apache.tomcat.dbcp.dbcp.DelegatingConnection.createStatement(DelegatingConnection.java:224)
>
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:331)
>
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312)
>
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.evict(GenericObjectPool.java:1217)
> - locked org.apache.tomcat.dbcp.pool.impl.GenericObjectPool@741a266
>
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1341)
> java.util.TimerThread.mainLoop(Unknown Source)
> java.util.TimerThread.run(Unknown Source)
>
>
> THREAD 2:
> Name: scheduling.QuartzInternal_Worker-0
> State: BLOCKED on
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool@741a266
> owned by: Timer-1
> Total blocked: 156,031 Total waited: 206,465
>
> Stack trace:
>
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.addObjectToPool(GenericObjectPool.java:1137)
>
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:1076)
>
> org.apache.tomcat.dbcp.dbcp.PoolableConnection.close(PoolableConnection.java:87)
> - locked org.apache.tomcat.dbcp.dbcp.PoolableConnection@1e667871
>
> org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
>
> org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:96)
>
> org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:451)
> org.hibernate.jdbc.ConnectionManager.cleanup(ConnectionManager.java:385)
> org.hibernate.jdbc.ConnectionManager.close(ConnectionManager.java:324)
> org.hibernate.impl.SessionImpl.close(SessionImpl.java:298)
>
> org.springframework.orm.hibernate3.SessionFactoryUtils.closeSession(SessionFactoryUtils.java:791)
>
> org.springframework.orm.hibernate3.SessionFactoryUtils.closeSessionOrRegisterDeferredClose(SessionFactoryUtils.java:777)
>
> org.springframework.orm.hibernate3.HibernateTransactionManager.doCleanupAfterCompletion(HibernateTransactionManager.java:733)
>
> org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:989)
>
> org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:782)
>
> org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
>
> org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
>
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
>
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
>
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> $Proxy8.execute(Unknown Source)
> sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> java.lang.reflect.Method.invoke(Unknown Source)
> org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:276)
>
> org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:260)
>
> org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
> org.quartz.core.JobRunShell.run(JobRunShell.java:203)
>
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
>
>
>
> Any help or insight into the problem and/or the correct dbcp settings would
> be appreciated to ensure that this doesn't happen again.
>
> Thanks,
>
> Eric
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
Re: dbcp pool evictor deadlock?
Posted by Mark Thomas <ma...@apache.org>.
Eric B. wrote:
>> "Caldarale, Charles R" <Ch...@unisys.com> wrote in message
>> news:0AAE5AB84B013E45A7B61CB66943C172294795F0D4@USEA-EXCH7.na.uis.unisys.com...
>>> From: news [mailto:news@ger.gmane.org] On Behalf Of Eric B.
>>> Subject: dbcp pool evictor deadlock?
>>>
>>> THREAD 1:
>>> Name: Timer-1
>>> State: BLOCKED on
>>> org.apache.tomcat.dbcp.dbcp.PoolableConnection@1e667871
>>> owned by: scheduling.QuartzInternal_Worker-0
>>> Total blocked: 1 Total waited: 15,342
>>>
>>> Stack trace:
>>> org.apache.tomcat.dbcp.dbcp.AbandonedTrace.addTrace(AbandonedTrace.java:175)
>> Assuming this version of Tomcat is using commons-dbcp 1.2.2, there's
>> something seriously wrong here. The synchronized block at the above line
>> is for an AbandonedTrace object, not a PoolableConnection.
Like a number of classes, PoolableConnection extends AbandonedTrace so
this isn't entirely unexpected.
Pool 1.5 fixed a bunch of sync issues. This looks like a variation of
POOL-125 / DBCP-44. Whilst not identical, the various changes in pool
1.5 should (hopefully) resolve this.
> Am running CentOS 5.2 on a 64bit server, running 64bit Sun Java 6.0.11 with
> Tomcat 6.0.18. I'm not sure what version of dbcp Tomcat 6.0.18 uses,
> however.
dbcp 1.2.2 / pool 1.4
> With respect to Mark's posting to switch to apache dpcp 1.2.2 and pool
> 1.5.1, is there any configuration documentation anywhere that I can read up
> to find out how to reconfigure tomcat/my app to use those instead of the
> default ones shipped with Tomcat? I figure I can d/l them and drop them in
> the tomcat/lib directory,
Correct.
> but not sure how to indicate to Tomcat to use
> those classes in the Resource tag.
Add a factory attribute to the resource tag of
org.apache.commons.dbcp.BasicDataSourceFactory
That should do it.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by "Eric B." <eb...@hotmail.com>.
>
> "Caldarale, Charles R" <Ch...@unisys.com> wrote in message
> news:0AAE5AB84B013E45A7B61CB66943C172294795F0D4@USEA-EXCH7.na.uis.unisys.com...
> > From: news [mailto:news@ger.gmane.org] On Behalf Of Eric B.
> > Subject: dbcp pool evictor deadlock?
> >
> > THREAD 1:
> > Name: Timer-1
> > State: BLOCKED on
> > org.apache.tomcat.dbcp.dbcp.PoolableConnection@1e667871
> > owned by: scheduling.QuartzInternal_Worker-0
> > Total blocked: 1 Total waited: 15,342
> >
> > Stack trace:
> > org.apache.tomcat.dbcp.dbcp.AbandonedTrace.addTrace(AbandonedTrace.java:175)
>
> Assuming this version of Tomcat is using commons-dbcp 1.2.2, there's
> something seriously wrong here. The synchronized block at the above line
> is for an AbandonedTrace object, not a PoolableConnection.
>
> This is beginning to look like a broken JVM or broken hardware. Want to
> tell us what you're using for those (and the OS, while you're at it)?
Am running CentOS 5.2 on a 64bit server, running 64bit Sun Java 6.0.11 with
Tomcat 6.0.18. I'm not sure what version of dbcp Tomcat 6.0.18 uses,
however.
With respect to Mark's posting to switch to apache dpcp 1.2.2 and pool
1.5.1, is there any configuration documentation anywhere that I can read up
to find out how to reconfigure tomcat/my app to use those instead of the
default ones shipped with Tomcat? I figure I can d/l them and drop them in
the tomcat/lib directory, but not sure how to indicate to Tomcat to use
those classes in the Resource tag.
Thanks!
Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: dbcp pool evictor deadlock?
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: news [mailto:news@ger.gmane.org] On Behalf Of Eric B.
> Subject: dbcp pool evictor deadlock?
>
> THREAD 1:
> Name: Timer-1
> State: BLOCKED on
> org.apache.tomcat.dbcp.dbcp.PoolableConnection@1e667871
> owned by: scheduling.QuartzInternal_Worker-0
> Total blocked: 1 Total waited: 15,342
>
> Stack trace:
> org.apache.tomcat.dbcp.dbcp.AbandonedTrace.addTrace(AbandonedTrace.java:175)
Assuming this version of Tomcat is using commons-dbcp 1.2.2, there's something seriously wrong here. The synchronized block at the above line is for an AbandonedTrace object, not a PoolableConnection.
This is beginning to look like a broken JVM or broken hardware. Want to tell us what you're using for those (and the OS, while you're at it)?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by Filip Hanik - Dev Lists <de...@hanik.com>.
Could be a bug in DBCP, sure looks like it.
You can try a drop in replacement that we are working on.
http://people.apache.org/~fhanik/jdbc-pool/v1.0.5/
Filip
Eric B. wrote:
> Hi,
>
> I appologize in advance for the cross-post, but I'm really not sure if this
> is a problem with the dbcp code or my configuration.
>
> I ran into the most troublesome deadlock over the weekend on my server, and
> cannot figure out how/or why this deadlock occured. Via jconsole, I was
> able to get thread information of the deadlock, but that hasn't helped me
> diagnose the problem any further and/or determine if it is a config error,
> or something more critical in the tomcat pool.
>
> >From what I can tell, there is a race condition where the evictor was
> triggered at the exact same moment as a resource was being added back into
> the pool. The evictor was waiting for the resource to be added to the pool,
> and the resource was waiting for the evictor to finish. However, I'm not
> sure if I am correct or not.
>
>
> My jndi settings are as follows:
> <!-- Database resource -->
> <Resource name="jdbc/db" auth="Container"
> type="javax.sql.DataSource" username="db" password="db"
> driverClassName="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/myapp?autoReconnect=true"
> maxActive="100" maxIdle="4"
> validationQuery="select 1"
> testOnBorrow="true"
> testWhileIdle="true"
> timeBetweenEvictionRunsMillis="10000"
> minEvictableIdleTimeMillis="60000" />
>
>
> My thread trace is the following:
> THREAD 1:
> Name: Timer-1
> State: BLOCKED on org.apache.tomcat.dbcp.dbcp.PoolableConnection@1e667871
> owned by: scheduling.QuartzInternal_Worker-0
> Total blocked: 1 Total waited: 15,342
>
> Stack trace:
> org.apache.tomcat.dbcp.dbcp.AbandonedTrace.addTrace(AbandonedTrace.java:175)
> org.apache.tomcat.dbcp.dbcp.AbandonedTrace.init(AbandonedTrace.java:92)
> org.apache.tomcat.dbcp.dbcp.AbandonedTrace.<init>(AbandonedTrace.java:82)
> org.apache.tomcat.dbcp.dbcp.DelegatingStatement.<init>(DelegatingStatement.java:61)
> org.apache.tomcat.dbcp.dbcp.DelegatingConnection.createStatement(DelegatingConnection.java:224)
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:331)
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:312)
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.evict(GenericObjectPool.java:1217)
> - locked org.apache.tomcat.dbcp.pool.impl.GenericObjectPool@741a266
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool$Evictor.run(GenericObjectPool.java:1341)
> java.util.TimerThread.mainLoop(Unknown Source)
> java.util.TimerThread.run(Unknown Source)
>
>
> THREAD 2:
> Name: scheduling.QuartzInternal_Worker-0
> State: BLOCKED on org.apache.tomcat.dbcp.pool.impl.GenericObjectPool@741a266
> owned by: Timer-1
> Total blocked: 156,031 Total waited: 206,465
>
> Stack trace:
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.addObjectToPool(GenericObjectPool.java:1137)
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:1076)
> org.apache.tomcat.dbcp.dbcp.PoolableConnection.close(PoolableConnection.java:87)
> - locked org.apache.tomcat.dbcp.dbcp.PoolableConnection@1e667871
> org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:181)
> org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.closeConnection(LocalDataSourceConnectionProvider.java:96)
> org.hibernate.jdbc.ConnectionManager.closeConnection(ConnectionManager.java:451)
> org.hibernate.jdbc.ConnectionManager.cleanup(ConnectionManager.java:385)
> org.hibernate.jdbc.ConnectionManager.close(ConnectionManager.java:324)
> org.hibernate.impl.SessionImpl.close(SessionImpl.java:298)
> org.springframework.orm.hibernate3.SessionFactoryUtils.closeSession(SessionFactoryUtils.java:791)
> org.springframework.orm.hibernate3.SessionFactoryUtils.closeSessionOrRegisterDeferredClose(SessionFactoryUtils.java:777)
> org.springframework.orm.hibernate3.HibernateTransactionManager.doCleanupAfterCompletion(HibernateTransactionManager.java:733)
> org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:989)
> org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:782)
> org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:701)
> org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:321)
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> $Proxy8.execute(Unknown Source)
> sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source)
> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> java.lang.reflect.Method.invoke(Unknown Source)
> org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:276)
> org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:260)
> org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
> org.quartz.core.JobRunShell.run(JobRunShell.java:203)
> org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)
>
>
>
> Any help or insight into the problem and/or the correct dbcp settings would
> be appreciated to ensure that this doesn't happen again.
>
> Thanks,
>
> Eric
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: dev-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by "Eric B." <eb...@hotmail.com>.
> > I ran into the most troublesome deadlock over the weekend
> > on my server
>
> Always start by telling us what version of Tomcat (and therefore which
> commons-dbcp) you're using. It's likely commons-dbcp 1.2.2 since that's
> been out for a while, but the exact version may be critical.
Sorry for the oversight. I'm running Tomcat 6.0.18.
Thanks,
Eric
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: dbcp pool evictor deadlock?
Posted by Mark Thomas <ma...@apache.org>.
Caldarale, Charles R wrote:
>> From: news [mailto:news@ger.gmane.org] On Behalf Of Eric B.
>> Subject: dbcp pool evictor deadlock?
>>
>> I ran into the most troublesome deadlock over the weekend
>> on my server
>
> Always start by telling us what version of Tomcat (and therefore which commons-dbcp) you're using. It's likely commons-dbcp 1.2.2 since that's been out for a while, but the exact version may be critical.
It may well be an issue with DBCP. Your best bet would be to switch your
app to use dbcp 1.2.2 and pool 1.5.1 directly rather than the built-in
version bundled with Tomcat.
There are some fixes needed in dbcp 1.2.2 but most of the sync issues
were in pool 1.4/
Plans are afoot to update Tomcat to these versions.
Mark
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: dbcp pool evictor deadlock?
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: news [mailto:news@ger.gmane.org] On Behalf Of Eric B.
> Subject: dbcp pool evictor deadlock?
>
> I ran into the most troublesome deadlock over the weekend
> on my server
Always start by telling us what version of Tomcat (and therefore which commons-dbcp) you're using. It's likely commons-dbcp 1.2.2 since that's been out for a while, but the exact version may be critical.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: dbcp pool evictor deadlock?
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: markt@apache.org
> Subject: RE: dbcp pool evictor deadlock?
> Like a number of classes, PoolableConnection extends
> AbandonedTrace so this isn't entirely unexpected.
I really shouldn't be looking at code with a splitting headache...
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org