You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Bruce Milner <br...@emberex.com> on 2018/09/07 21:56:23 UTC
[pool] Possible deadlock or user error (me) commons-pool 2.5.0
Hello,
I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
for not using out-of-the-box is that the existing code relies on
changing catalogs at runtime reusing an existing connection. The
original design was to use multiple databases using the same connection
and this cannot be changed.
I recently replaced a lot of hand crafted code with the commons-pool2
implementation.
The issue I have is that one server I manage went into a state where
there are plenty of connections, but none are being returned to the
pool. They are all stuck on a lock inside of
GenericKeyedObjectPool.returnObject.
The config is basically
GenericKeyedObjectPoolConfig config = new
GenericKeyedObjectPoolConfig();
config.setBlockWhenExhausted(true);
config.setMaxTotal(120);
config.setMaxTotalPerKey(60);
config.setTestOnBorrow(true);
config.setTimeBetweenEvictionRunsMillis(60000);
config.setMinEvictableIdleTimeMillis(0); // don't starve
connections because of catalog switches.
/**
* For database connections, use FIFO so that we get rid of
older connections first before newer ones.
*/
config.setLifo(false);
return new GenericKeyedObjectPool<String, PooledConnection>(new
PooledConnectionFactory(), config);
There are 150 of these threads waiting on a lock to release connections
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
at
com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
and 158 of these threads waiting to open connections.
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
at
org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
at
com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
For the "- parking to wait for <0x00000006471cd7d8> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
Though I see no other messages with that object in the thread dump.
Has anyone run into this? It seems like some sort of deadlock.
--
Bruce Milner
Senior Software Developer (Emberex)
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
Hello Bruce,
This sounds a bit like a discussion we had about missing wakeups. I think it’s was related to depleted pools. Didn’t find the discussion, hopefully somebody else recalls the conditions? I think it was not fixed.
Gruss
Bernd
--
http://bernd.eckenfels.net
________________________________
Von: Bruce Milner <br...@emberex.com>
Gesendet: Samstag, September 8, 2018 8:19 PM
An: user@commons.apache.org
Betreff: Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Hello,
I did a while back, but my understanding of DBCP is that it has one pool
per database and we have thousands.
With the number of nodes serving the application multiplied by the
number of databases, it could easily exceed maximum number of
connections to existing SQL database server. The individual databases
are mostly shared by one database server each have individual schemas.
We keep track of the database URL and switch the connection via
connector.setCatalog(). We also have some that co-exist, so the
connection pool has the smarts to decide if it needs a catalog change.
The commons pool has been working great so far, and this is the only
case I have see where we ended up in this state. We haven't seen this
with load tests, but this once in production.
I was hoping if this exposed a bug, could get fixed in the pool code. I
don't have a reproduction case at this time. I forgot to mention that
the environment is java 8 141 with Connector/J 5.1.45
--bruce
On 9/7/2018 5:05 PM, Gary Gregory wrote:
> Hi,
>
> A side question: Have you tried Apache Commons DBCP (which is based on
> Commons Pool)?
>
> https://commons.apache.org/proper/commons-dbcp/
>
> Gary
>
> On Fri, Sep 7, 2018 at 5:24 PM Bruce Milner <br...@emberex.com>
> wrote:
>
>> Hello,
>>
>> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
>> for not using out-of-the-box is that the existing code relies on
>> changing catalogs at runtime reusing an existing connection. The
>> original design was to use multiple databases using the same connection
>> and this cannot be changed.
>>
>> I recently replaced a lot of hand crafted code with the commons-pool2
>> implementation.
>>
>> The issue I have is that one server I manage went into a state where
>> there are plenty of connections, but none are being returned to the
>> pool. They are all stuck on a lock inside of
>> GenericKeyedObjectPool.returnObject.
>>
>> The config is basically
>> GenericKeyedObjectPoolConfig config = new
>> GenericKeyedObjectPoolConfig();
>> config.setBlockWhenExhausted(true);
>> config.setMaxTotal(120);
>> config.setMaxTotalPerKey(60);
>> config.setTestOnBorrow(true);
>> config.setTimeBetweenEvictionRunsMillis(60000);
>> config.setMinEvictableIdleTimeMillis(0); // don't starve
>> connections because of catalog switches.
>> /**
>> * For database connections, use FIFO so that we get rid of
>> older connections first before newer ones.
>> */
>> config.setLifo(false);
>> return new GenericKeyedObjectPool<String, PooledConnection>(new
>> PooledConnectionFactory(), config);
>>
>> There are 150 of these threads waiting on a lock to release connections
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>> at
>>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>> at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>> at
>>
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>>
>> and 158 of these threads waiting to open connections.
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>> at
>>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>> at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>> at
>>
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
>> at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>>
>> For the "- parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>> Though I see no other messages with that object in the thread dump.
>>
>> Has anyone run into this? It seems like some sort of deadlock.
>>
>> --
>> Bruce Milner
>> Senior Software Developer (Emberex)
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
>>
--
Bruce Milner
Senior Software Developer (Emberex)
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Posted by Bruce Milner <br...@emberex.com>.
Hello,
I did a while back, but my understanding of DBCP is that it has one pool
per database and we have thousands.
With the number of nodes serving the application multiplied by the
number of databases, it could easily exceed maximum number of
connections to existing SQL database server. The individual databases
are mostly shared by one database server each have individual schemas.
We keep track of the database URL and switch the connection via
connector.setCatalog(). We also have some that co-exist, so the
connection pool has the smarts to decide if it needs a catalog change.
The commons pool has been working great so far, and this is the only
case I have see where we ended up in this state. We haven't seen this
with load tests, but this once in production.
I was hoping if this exposed a bug, could get fixed in the pool code. I
don't have a reproduction case at this time. I forgot to mention that
the environment is java 8 141 with Connector/J 5.1.45
--bruce
On 9/7/2018 5:05 PM, Gary Gregory wrote:
> Hi,
>
> A side question: Have you tried Apache Commons DBCP (which is based on
> Commons Pool)?
>
> https://commons.apache.org/proper/commons-dbcp/
>
> Gary
>
> On Fri, Sep 7, 2018 at 5:24 PM Bruce Milner <br...@emberex.com>
> wrote:
>
>> Hello,
>>
>> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
>> for not using out-of-the-box is that the existing code relies on
>> changing catalogs at runtime reusing an existing connection. The
>> original design was to use multiple databases using the same connection
>> and this cannot be changed.
>>
>> I recently replaced a lot of hand crafted code with the commons-pool2
>> implementation.
>>
>> The issue I have is that one server I manage went into a state where
>> there are plenty of connections, but none are being returned to the
>> pool. They are all stuck on a lock inside of
>> GenericKeyedObjectPool.returnObject.
>>
>> The config is basically
>> GenericKeyedObjectPoolConfig config = new
>> GenericKeyedObjectPoolConfig();
>> config.setBlockWhenExhausted(true);
>> config.setMaxTotal(120);
>> config.setMaxTotalPerKey(60);
>> config.setTestOnBorrow(true);
>> config.setTimeBetweenEvictionRunsMillis(60000);
>> config.setMinEvictableIdleTimeMillis(0); // don't starve
>> connections because of catalog switches.
>> /**
>> * For database connections, use FIFO so that we get rid of
>> older connections first before newer ones.
>> */
>> config.setLifo(false);
>> return new GenericKeyedObjectPool<String, PooledConnection>(new
>> PooledConnectionFactory(), config);
>>
>> There are 150 of these threads waiting on a lock to release connections
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>> at
>>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>> at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>> at
>>
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>>
>> and 158 of these threads waiting to open connections.
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>> at
>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>> at
>>
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>> at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>> at
>>
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
>> at
>>
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
>> at
>>
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
>> at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>>
>> For the "- parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>> Though I see no other messages with that object in the thread dump.
>>
>> Has anyone run into this? It seems like some sort of deadlock.
>>
>> --
>> Bruce Milner
>> Senior Software Developer (Emberex)
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
>> For additional commands, e-mail: user-help@commons.apache.org
>>
>>
--
Bruce Milner
Senior Software Developer (Emberex)
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Posted by Gary Gregory <ga...@gmail.com>.
Hi,
A side question: Have you tried Apache Commons DBCP (which is based on
Commons Pool)?
https://commons.apache.org/proper/commons-dbcp/
Gary
On Fri, Sep 7, 2018 at 5:24 PM Bruce Milner <br...@emberex.com>
wrote:
> Hello,
>
> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
> for not using out-of-the-box is that the existing code relies on
> changing catalogs at runtime reusing an existing connection. The
> original design was to use multiple databases using the same connection
> and this cannot be changed.
>
> I recently replaced a lot of hand crafted code with the commons-pool2
> implementation.
>
> The issue I have is that one server I manage went into a state where
> there are plenty of connections, but none are being returned to the
> pool. They are all stuck on a lock inside of
> GenericKeyedObjectPool.returnObject.
>
> The config is basically
> GenericKeyedObjectPoolConfig config = new
> GenericKeyedObjectPoolConfig();
> config.setBlockWhenExhausted(true);
> config.setMaxTotal(120);
> config.setMaxTotalPerKey(60);
> config.setTestOnBorrow(true);
> config.setTimeBetweenEvictionRunsMillis(60000);
> config.setMinEvictableIdleTimeMillis(0); // don't starve
> connections because of catalog switches.
> /**
> * For database connections, use FIFO so that we get rid of
> older connections first before newer ones.
> */
> config.setLifo(false);
> return new GenericKeyedObjectPool<String, PooledConnection>(new
> PooledConnectionFactory(), config);
>
> There are 150 of these threads waiting on a lock to release connections
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
>
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
>
> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
> at
>
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
> at
>
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
> at
>
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
> at
>
> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
> at
>
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>
> and 158 of these threads waiting to open connections.
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
> at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
> at
>
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
>
> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
> at
>
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
> at
>
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
> at
>
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
> at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>
> For the "- parking to wait for <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
> Though I see no other messages with that object in the thread dump.
>
> Has anyone run into this? It seems like some sort of deadlock.
>
> --
> Bruce Milner
> Senior Software Developer (Emberex)
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Posted by Phil Steitz <ph...@gmail.com>.
On 9/9/18 12:16 AM, Mark Thomas wrote:
> On 08/09/18 19:36, Mark Thomas wrote:
>
> <snip/>
>
>>> For the "- parking to wait for <0x00000006471cd7d8> (a
>>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>>> Though I see no other messages with that object in the thread dump.
>>>
>>> Has anyone run into this? It seems like some sort of deadlock.
>> Do you still have the full thread dump? Can you post it somewhere (where
>> we can look at it)?
> Thanks. Lots of threads waiting for the lock. None holding it. That is
> very strange.
>
> Some variation of this?
>
> https://confluence.atlassian.com/jirakb/jira-applications-stall-due-to-stackoverflowerror-exception-941601100.html
>
Here is one thing to look at. Note that the take request from
borrowObject comes from
p = objectDeque.getIdleObjects().pollFirst();
This LinkedBlockingDeque method does not register the thread as
waiting on the notEmpty condition in LinkedBlockingDeque the way
that say
p = objectDeque.getIdleObjects().pollFirst(
borrowMaxWaitMillis,
TimeUnit.MILLISECONDS);
does.
Phil
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Posted by Bruce Milner <br...@emberex.com>.
On 9/9/2018 12:16 AM, Mark Thomas wrote:
> On 08/09/18 19:36, Mark Thomas wrote:
>
> <snip/>
>
>>> For the "- parking to wait for <0x00000006471cd7d8> (a
>>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>>> Though I see no other messages with that object in the thread dump.
>>>
>>> Has anyone run into this? It seems like some sort of deadlock.
>> Do you still have the full thread dump? Can you post it somewhere (where
>> we can look at it)?
> Thanks. Lots of threads waiting for the lock. None holding it. That is
> very strange.
>
> Some variation of this?
>
> https://confluence.atlassian.com/jirakb/jira-applications-stall-due-to-stackoverflowerror-exception-941601100.html
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
Yeah, I looked closer at thread dump and no thread holding lock. Also
looked in past several months of logs and no StackOverflowError's .
My guess now is JVM bug.
Thanks for looking!
--bruce
--
Bruce Milner
Senior Software Developer (Emberex)
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Posted by Mark Thomas <ma...@apache.org>.
On 08/09/18 19:36, Mark Thomas wrote:
<snip/>
>> For the "- parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>> Though I see no other messages with that object in the thread dump.
>>
>> Has anyone run into this? It seems like some sort of deadlock.
>
> Do you still have the full thread dump? Can you post it somewhere (where
> we can look at it)?
Thanks. Lots of threads waiting for the lock. None holding it. That is
very strange.
Some variation of this?
https://confluence.atlassian.com/jirakb/jira-applications-stall-due-to-stackoverflowerror-exception-941601100.html
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Posted by Bruce Milner <br...@emberex.com>.
Hello Mark,
I sent the thread dump directly to you.
On 9/8/2018 11:36 AM, Mark Thomas wrote:
> On 07/09/18 22:56, Bruce Milner wrote:
>> Hello,
>>
>> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
>> for not using out-of-the-box is that the existing code relies on
>> changing catalogs at runtime reusing an existing connection. The
>> original design was to use multiple databases using the same connection
>> and this cannot be changed.
>>
>> I recently replaced a lot of hand crafted code with the commons-pool2
>> implementation.
>>
>> The issue I have is that one server I manage went into a state where
>> there are plenty of connections, but none are being returned to the
>> pool. They are all stuck on a lock inside of
>> GenericKeyedObjectPool.returnObject.
>>
>> The config is basically
>> GenericKeyedObjectPoolConfig config = new
>> GenericKeyedObjectPoolConfig();
>> config.setBlockWhenExhausted(true);
>> config.setMaxTotal(120);
>> config.setMaxTotalPerKey(60);
>> config.setTestOnBorrow(true);
>> config.setTimeBetweenEvictionRunsMillis(60000);
>> config.setMinEvictableIdleTimeMillis(0); // don't starve
>> connections because of catalog switches.
>> /**
>> * For database connections, use FIFO so that we get rid of
>> older connections first before newer ones.
>> */
>> config.setLifo(false);
>> return new GenericKeyedObjectPool<String, PooledConnection>(new
>> PooledConnectionFactory(), config);
>>
>> There are 150 of these threads waiting on a lock to release connections
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>>
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>>
>> at
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>>
>> at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>> at
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
>>
>> at
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
>>
>> at
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
>>
>> at
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
>>
>> at
>> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
>>
>> at
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>>
>>
>> and 158 of these threads waiting to open connections.
>> java.lang.Thread.State: WAITING (parking)
>> at sun.misc.Unsafe.park(Native Method)
>> - parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>>
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>>
>> at
>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>>
>> at
>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>>
>> at
>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
>> at
>> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
>>
>> at
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
>>
>> at
>> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
>>
>> at
>> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
>>
>> at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>>
>> For the "- parking to wait for <0x00000006471cd7d8> (a
>> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
>> Though I see no other messages with that object in the thread dump.
>>
>> Has anyone run into this? It seems like some sort of deadlock.
> Do you still have the full thread dump? Can you post it somewhere (where
> we can look at it)?
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
--
Bruce Milner
Senior Software Developer (Emberex)
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [pool] Possible deadlock or user error (me) commons-pool 2.5.0
Posted by Mark Thomas <ma...@apache.org>.
On 07/09/18 22:56, Bruce Milner wrote:
> Hello,
>
> I am using commons-pool2-2.5.0 for a MySQL connection pooler. The reason
> for not using out-of-the-box is that the existing code relies on
> changing catalogs at runtime reusing an existing connection. The
> original design was to use multiple databases using the same connection
> and this cannot be changed.
>
> I recently replaced a lot of hand crafted code with the commons-pool2
> implementation.
>
> The issue I have is that one server I manage went into a state where
> there are plenty of connections, but none are being returned to the
> pool. They are all stuck on a lock inside of
> GenericKeyedObjectPool.returnObject.
>
> The config is basically
> GenericKeyedObjectPoolConfig config = new
> GenericKeyedObjectPoolConfig();
> config.setBlockWhenExhausted(true);
> config.setMaxTotal(120);
> config.setMaxTotalPerKey(60);
> config.setTestOnBorrow(true);
> config.setTimeBetweenEvictionRunsMillis(60000);
> config.setMinEvictableIdleTimeMillis(0); // don't starve
> connections because of catalog switches.
> /**
> * For database connections, use FIFO so that we get rid of
> older connections first before newer ones.
> */
> config.setLifo(false);
> return new GenericKeyedObjectPool<String, PooledConnection>(new
> PooledConnectionFactory(), config);
>
> There are 150 of these threads waiting on a lock to release connections
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.hasTakeWaiters(LinkedBlockingDeque.java:1389)
>
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.hasBorrowWaiters(GenericKeyedObjectPool.java:849)
>
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.returnObject(GenericKeyedObjectPool.java:551)
>
> at
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.releaseConnection(ConnectionPooler.java:358)
>
> at
> com.ilrn.util.sql.connectionpooler.PooledConnection.close(PooledConnection.java:141)
>
> at
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.safeClose(ConnectionPooler.java:480)
>
>
> and 158 of these threads waiting to open connections.
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>
> at
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
>
> at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
> at
> org.apache.commons.pool2.impl.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:560)
>
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356)
>
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
>
> at
> com.ilrn.util.sql.connectionpooler.ConnectionPooler.getConnection(ConnectionPooler.java:197)
>
> at com.ilrn.util.sql.Database.getConnection(Database.java:1273)
>
> For the "- parking to wait for <0x00000006471cd7d8> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)" there are 155.
> Though I see no other messages with that object in the thread dump.
>
> Has anyone run into this? It seems like some sort of deadlock.
Do you still have the full thread dump? Can you post it somewhere (where
we can look at it)?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org