You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mikolaj Rydzewski <mi...@ceti.pl> on 2008/11/07 11:47:17 UTC

Deadlocks with Oracle

Hi,

I have following setup: Tomcat 5.5.23 with Oracle database 10.2.0.1.0 
with DataSource defined as:

<Resource name="jdbc/db" auth="Container" type="javax.sql.DataSource"
    maxActive="50" maxIdle="10" maxWait="10000"
    validationQuery="select sysdate from dual" testOnBorrow="true"
    username="x" password="x"
    driverClassName="oracle.jdbc.OracleDriver"
    url="jdbc:oracle:thin:@10.76.162.21:1532:CAP"
    removeAbandoned="true" removeAbandonedTimeout="30"
    logAbandoned="true" />

There are several webapps deployed. After some time, depending on usage, 
applications get stucked - I mean every request that needs database 
connection hangs. Thread dump looks ALWAYS the same:

"http-8080-Processor21" daemon prio=1 tid=0x22aed1d8 nid=0x6958 runnable 
[0x20a6c000..0x20a6e7f0]
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at oracle.net.ns.Packet.receive(Unknown Source)
    at oracle.net.ns.DataPacket.receive(Unknown Source)
    at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
    at oracle.net.ns.NetInputStream.read(Unknown Source)
    at oracle.net.ns.NetInputStream.read(Unknown Source)
    at oracle.net.ns.NetInputStream.read(Unknown Source)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1099)
    at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
    at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:478)
    at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
    at 
oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:790)
    at 
oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1039) 

    at 
oracle.jdbc.driver.T4CStatement.executeMaybeDescribe(T4CStatement.java:830)
    at 
oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1132) 

    at 
oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1272)
    - locked <0xac40fc70> (a oracle.jdbc.driver.T4CStatement)
    - locked <0x90979730> (a oracle.jdbc.driver.T4CConnection)
    at 
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:205) 

    at 
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:328) 

    at 
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:308) 

    at 
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:788) 

    at 
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) 

    at 
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) 

    at 
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) 


It happens ONLY on this particular environment/setup. So I doubt it's 
application's fault.

For testing purposes I have deployed simply webapp, that on request 
calls DataSource.getConnection() in a loop until exception occurs. Then 
it closes every achieved connection. Just for test. And to my surprise 
it helps!
When other apps hang waiting for Connections (confirmed with thread 
dump), I issue a request to my testing app, it starts to iterate, then 
'org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot get a 
connection, pool exhausted' occurs and everything becomes normal. 
Blocked threads are no longer blocked, users can continue their work.

Has anyone got any clues? What and where is wrong?

-- 
Mikolaj Rydzewski <mi...@ceti.pl>


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Deadlocks with Oracle

Posted by Juha Laiho <Ju...@iki.fi>.
Mikolaj Rydzewski wrote:
> I have following setup: Tomcat 5.5.23 with Oracle database 10.2.0.1.0
...
> There are several webapps deployed. After some time, depending on usage,
> applications get stucked - I mean every request that needs database
> connection hangs. Thread dump looks ALWAYS the same:
> 
> "http-8080-Processor21" daemon prio=1 tid=0x22aed1d8 nid=0x6958 runnable
> [0x20a6c000..0x20a6e7f0]
...
>    at
> oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1272)
>    - locked <0xac40fc70> (a oracle.jdbc.driver.T4CStatement)
>    - locked <0x90979730> (a oracle.jdbc.driver.T4CConnection)
...

I may have seen something similar -- however unlike in the case you described,
the only way we could recover was to restart Tomcat. However, once we upgraded
to a more recent Oracle JDBC driver, the problem was gone (there are one-off
Oracle patches that do update the JDBC driver, compared to the stock version;
the driver we're using now is a result of installing such a patch; I'm rather
certain the patches are included in 10.2.0.3 patchset). Getting these does
require a valid support contract, though.

Apologies for the vagueness of details (and I'm afraid I'm unable to locate
the details later, either), it's some time since I was wrestling with this.

But in any case, Oracle JDBC drivers seem to have bugs in the early versions
(you're quoting db 10.2.0.1.0, thus an early version of 10gRel2; I suppose
your JDBC driver comes from that installation).
-- 
..Juha

---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Deadlocks with Oracle

Posted by Mikolaj Rydzewski <mi...@ceti.pl>.
Martin Gainty wrote:
> Are you using the DBCP 1.2.2 distro as Filip suggested?
>   
Filip? Am I missing any emails from list? I use Tomcat 5.5.23.

-- 
Mikolaj Rydzewski <mi...@ceti.pl>


---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

RE: Deadlocks with Oracle

Posted by Martin Gainty <mg...@hotmail.com>.
Mikolaj

Are you using the DBCP 1.2.2 distro as Filip suggested?

Dziekuje/
Martin 
______________________________________________ 
Disclaimer and confidentiality note 
Everything in this e-mail and any attachments relates to the official business of Sender. This transmission is of a confidential nature and Sender does not endorse distribution to any party other than intended recipient. Sender does not necessarily endorse content contained within this transmission. 




> Date: Fri, 7 Nov 2008 11:47:17 +0100
> From: miki@ceti.pl
> To: users@tomcat.apache.org
> Subject: Deadlocks with Oracle
> 
> Hi,
> 
> I have following setup: Tomcat 5.5.23 with Oracle database 10.2.0.1.0 
> with DataSource defined as:
> 
> <Resource name="jdbc/db" auth="Container" type="javax.sql.DataSource"
>     maxActive="50" maxIdle="10" maxWait="10000"
>     validationQuery="select sysdate from dual" testOnBorrow="true"
>     username="x" password="x"
>     driverClassName="oracle.jdbc.OracleDriver"
>     url="jdbc:oracle:thin:@10.76.162.21:1532:CAP"
>     removeAbandoned="true" removeAbandonedTimeout="30"
>     logAbandoned="true" />
> 
> There are several webapps deployed. After some time, depending on usage, 
> applications get stucked - I mean every request that needs database 
> connection hangs. Thread dump looks ALWAYS the same:
> 
> "http-8080-Processor21" daemon prio=1 tid=0x22aed1d8 nid=0x6958 runnable 
> [0x20a6c000..0x20a6e7f0]
>     at java.net.SocketInputStream.socketRead0(Native Method)
>     at java.net.SocketInputStream.read(SocketInputStream.java:129)
>     at oracle.net.ns.Packet.receive(Unknown Source)
>     at oracle.net.ns.DataPacket.receive(Unknown Source)
>     at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
>     at oracle.net.ns.NetInputStream.read(Unknown Source)
>     at oracle.net.ns.NetInputStream.read(Unknown Source)
>     at oracle.net.ns.NetInputStream.read(Unknown Source)
>     at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1099)
>     at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1070)
>     at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:478)
>     at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:207)
>     at 
> oracle.jdbc.driver.T4CStatement.executeForDescribe(T4CStatement.java:790)
>     at 
> oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:1039) 
> 
>     at 
> oracle.jdbc.driver.T4CStatement.executeMaybeDescribe(T4CStatement.java:830)
>     at 
> oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1132) 
> 
>     at 
> oracle.jdbc.driver.OracleStatement.executeQuery(OracleStatement.java:1272)
>     - locked <0xac40fc70> (a oracle.jdbc.driver.T4CStatement)
>     - locked <0x90979730> (a oracle.jdbc.driver.T4CConnection)
>     at 
> org.apache.tomcat.dbcp.dbcp.DelegatingStatement.executeQuery(DelegatingStatement.java:205) 
> 
>     at 
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateConnection(PoolableConnectionFactory.java:328) 
> 
>     at 
> org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.validateObject(PoolableConnectionFactory.java:308) 
> 
>     at 
> org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:788) 
> 
>     at 
> org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) 
> 
>     at 
> org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) 
> 
>     at 
> org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) 
> 
> 
> It happens ONLY on this particular environment/setup. So I doubt it's 
> application's fault.
> 
> For testing purposes I have deployed simply webapp, that on request 
> calls DataSource.getConnection() in a loop until exception occurs. Then 
> it closes every achieved connection. Just for test. And to my surprise 
> it helps!
> When other apps hang waiting for Connections (confirmed with thread 
> dump), I issue a request to my testing app, it starts to iterate, then 
> 'org.apache.tomcat.dbcp.dbcp.SQLNestedException: Cannot get a 
> connection, pool exhausted' occurs and everything becomes normal. 
> Blocked threads are no longer blocked, users can continue their work.
> 
> Has anyone got any clues? What and where is wrong?
> 
> -- 
> Mikolaj Rydzewski <mi...@ceti.pl>
> 
> 
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
> 

_________________________________________________________________
Windows Live Hotmail now works up to 70% faster.
http://windowslive.com/Explore/Hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_faster_112008