DO NOT REPLY [Bug 22229] New: - Foul connection causes livelock of all pool operations

[bu...@apache.org]

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://nagoya.apache.org/bugzilla/show_bug.cgi?id=22229>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=22229

Foul connection causes livelock of all pool operations

           Summary: Foul connection causes livelock of all pool operations
           Product: Commons
           Version: 1.0 Final
          Platform: All
        OS/Version: All
            Status: NEW
          Severity: Critical
          Priority: Other
         Component: Dbcp
        AssignedTo: commons-dev@jakarta.apache.org
        ReportedBy: Alfred.Nathaniel@swx.com


We have a configuration where it can happen that opening a new database 
connection fouls up and gets stuck waiting for a long timeout:

Thread-2288:
  [1] oracle.jdbc.oci8.OCIDBAccess.do_open (native method)
  [2] oracle.jdbc.oci8.OCIDBAccess.logon (OCIDBAccess.java:359)
  [3] oracle.jdbc.driver.OracleConnection.<init> (OracleConnection.java:252)
  [4] oracle.jdbc.driver.OracleDriver.getConnectionInstance 
(OracleDriver.java:365)
  [5] oracle.jdbc.driver.OracleDriver.connect (OracleDriver.java:260)
  [6] org.apache.commons.dbcp.DriverConnectionFactory.createConnection 
(DriverConnectionFactory.java:83)
  [7] org.apache.commons.dbcp.PoolableConnectionFactory.makeObject 
(PoolableConnectionFactory.java:184)
  [8] org.apache.commons.pool.impl.GenericObjectPool.borrowObject (null)
  [9] org.apache.commons.dbcp.AbandonedObjectPool.borrowObject 
(AbandonedObjectPool.java:117)
  [10] org.apache.commons.dbcp.PoolingDataSource.getConnection 
(PoolingDataSource.java:110)
  [11] org.apache.commons.dbcp.BasicDataSource.getConnection 
(BasicDataSource.java:312)
  [12] swx.eservices.db.DbManager.getConnection (DbManager.java:197)
...

The problem is that during that time the foul thread holds the lock on the 
AbandonedObjectPool that all other threads doing returnObject or borrowObject 
queue:

Thread-2278:
  [1] org.apache.commons.dbcp.AbandonedObjectPool.returnObject 
(AbandonedObjectPool.java:137)
  [2] org.apache.commons.dbcp.PoolableConnection.close 
(PoolableConnection.java:110)
  [3] swx.eservices.db.DbTransaction.commit (DbTransaction.java:158)
...

So perfectly working pooled connections can no longer be returned or borrowed, 
therefore freezing all threads with database access.  One foul apple makes the 
full basket rot.

If one waits long enough (30 minutes) the do_open will finally timeout and 
everything will continue except for the failed connection.  But of course that 
timeout is far to long to go unnoticed on a webserver.

What needs to be done is to open a new connection (or do the validationQuery) 
without holding the lock on the pool.  NB borrow/returnObject in 
dbcp/AbandonedObjectPool.java was made synchronized just before the DBCP_1_0 to 
fix a threading bug (commit comment to r1.2).