TP-Processor threads "waiting for monitor entry" on DBCP

[Isaac Arias <to...@spamgourmet.com>]

Hi All,

We're running multiple Tomcats (4.1.30) on Linux (2.4.21) fronted by 
different versions of Apache HTTP servers (1.3 and 2.0 flavors) using 
mod_jk and mod_jk2. These servers handle upwards of 60 Million pages a 
day. The app uses JDBC to talk to an Oracle database through the Tomcat 
DBCP connection pool.

After some time of high-usage, some of these Tomcats start hanging JK 
connections from the web servers, eventually consuming them all, 
therefore hanging the web servers as well.

Dumping the thread stack traces on these problematic Tomcats show 
several threads locked in a particular DBCP call:

"TP-Processor19" daemon prio=1 tid=0x08075a90 nid=0x22ec waiting for 
monitor entry [7beea000..7beebcd0]
        at 
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:300)
        - waiting to lock <0x85336b60> (a 
org.apache.commons.dbcp.PoolableConnectionFactory)
        at 
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:816)
        at 
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
        at 
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)
        ...
After a while:

"TP-Processor19" daemon prio=1 tid=0x08075a90 nid=0x22ec waiting for 
monitor entry [7beea000..7beebcd0]
        at 
org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:300)
        - waiting to lock <0x8527d4c8> (a 
org.apache.commons.dbcp.PoolableConnectionFactory)

Note that the same thread is still waiting. However, the makeObject() 
call reports that it's waiting to lock on some other number. I don't 
know if this is meaningful.

Later, additional threads appear as waiting on a different call:

"TP-Processor351" daemon prio=1 tid=0x79106258 nid=0x26c0 in 
Object.wait() [6dadf000..6dadfcd0]
        at java.lang.Object.wait(Native Method)
        - waiting on <0x8527aab8> (a 
org.apache.commons.pool.impl.GenericObjectPool)
        at java.lang.Object.wait(Object.java:429)
        at 
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:793)
        - locked <0x8527aab8> (a 
org.apache.commons.pool.impl.GenericObjectPool)
        at 
org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:140)
        at 
org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:518)

At first we thought this could be related to the Linux NPTL - JVM 
threading bug and set the env variable LD_ASSUME_KERNEL=2.4.19, but this 
didn't help. We have upgraded to the latest 1.4 JVM (1.4.2_07) and we're 
still seeing the problem.

Our code has been carefully audited twice and absolutely all ResultSets 
and Statements are closed after usage and all connections properly 
returned to the pool.

This happens predictably (although not periodically) when there are a 
high number (> 1.500) of concurrent sessions on a single Tomcat instance 
over a few minutes/hours.

Has anybody else seen these locks? Are we missing something obvious 
about DBCP configuration and usage?

Thanks for your help,

Ike

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org