You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Colin Ingarfield <co...@gmail.com> on 2013/03/12 22:47:47 UTC
Deadlock when using jetty 8 JDBCSessionManager and Tomcat 7 JDBC Connector
Hello,
We're using Jetty 8.1.3.v20120416 w/ JDBCSessionManager as our application
server with Tomcat 7's JDBC Connection pool 7.0.28. We've run this
particular combination in production since at least Sept 2012. (Java 6 64
bit/Ubuntu, Amazon RDS/MySQL db.)
This morning one Jetty instance stopped accepting requests and the problem
appears to be a deadlock btw a jetty connection handler thread and the conn
pool's "PoolCleaner" thread. From jstack trace:
Deadlock Detection:
Found one Java-level deadlock:
=============================
"qtp1840392480-3740":
waiting to lock Monitor@0x00007f4350001fd0 (Object@0x00000006c01a0e88, a
com/mysql/jdbc/JDBC4Connection),
which is held by "PoolCleaner[2009981184:1363034108768]"
"PoolCleaner[2009981184:1363034108768]":
waiting to lock Monitor@0x00007f4350001f28 (Object@0x00000006c1ed5738, a
com/mysql/jdbc/JDBC4ResultSet),
which is held by "qtp1840392480-3740"
Once this happened all other worker threads blocked at various points
within JDBCSession manager. Obviously restarting the instance resolved the
issue for now.
How does one go about diagnosing / resolving such an issue? It's not clear
to me if either component is "at fault" here...I have the full jstack
output as well as a full heap dump of the java process, if that would help.
Regards,
Colin
Re: Deadlock when using jetty 8 JDBCSessionManager and Tomcat 7 JDBC Connector
Posted by "Howard W. Smith, Jr." <sm...@gmail.com>.
On Tue, Mar 12, 2013 at 5:51 PM, Mark Thomas <ma...@apache.org> wrote:
> On 12/03/2013 21:47, Colin Ingarfield wrote:
> > Hello,
> >
> > We're using Jetty 8.1.3.v20120416 w/ JDBCSessionManager as our
> application
> > server with Tomcat 7's JDBC Connection pool 7.0.28. We've run this
> > particular combination in production since at least Sept 2012. (Java 6
> 64
> > bit/Ubuntu, Amazon RDS/MySQL db.)
> >
> > This morning one Jetty instance stopped accepting requests and the
> problem
> > appears to be a deadlock btw a jetty connection handler thread and the
> conn
> > pool's "PoolCleaner" thread. From jstack trace:
> > Deadlock Detection:
> >
> > Found one Java-level deadlock:
> > =============================
> >
> > "qtp1840392480-3740":
> > waiting to lock Monitor@0x00007f4350001fd0 (Object@0x00000006c01a0e88,
> a
> > com/mysql/jdbc/JDBC4Connection),
> > which is held by "PoolCleaner[2009981184:1363034108768]"
> > "PoolCleaner[2009981184:1363034108768]":
> > waiting to lock Monitor@0x00007f4350001f28 (Object@0x00000006c1ed5738,
> a
> > com/mysql/jdbc/JDBC4ResultSet),
> > which is held by "qtp1840392480-3740"
> >
> > Once this happened all other worker threads blocked at various points
> > within JDBCSession manager. Obviously restarting the instance resolved
> the
> > issue for now.
> >
> > How does one go about diagnosing / resolving such an issue? It's not
> clear
> > to me if either component is "at fault" here...I have the full jstack
> > output as well as a full heap dump of the java process, if that would
> help.
>
> Deadlocks occur when two threads attempt to acquire the same locks but
> in a different order. Normally, one of the code paths is at fault. To
> figure out which we need to see the full stack trace for the two threads
> that are deadlocked.
>
> Mark
>
>
I experienced deadlock today while revisiting development of some code that
would automatically insert some data in the background via @Schedule timers.
While first testing the code, I had CDI @RequestScoped bean referencing
@Stateless EJBs to perform inserts against many different tables. That is
when I experienced the deadlock, but then I changed the bean from CDI
@RequestScoped back to @Singleton and @Lock(WRITE) (as advised by David
Blevins of TomEE/openEJB...months ago), and immediately the deadlock went
away. Using TomEE (tomcat7), ActiveMQ/JMS, @Schedule, @Singleton
@Lock(WRITE) and plenty of @Stateless EJBs (1 per table) to get it all
done.... and on my development server, takes few seconds for everything to
work, and 1 second on the database insert and commit.
I assume that the CDI @RequestScoped and @Stateless EJBs were executing
concurrently/simultaneously, which I assume is the 2 threads that Mark was
talking about here.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Deadlock when using jetty 8 JDBCSessionManager and Tomcat 7 JDBC
Connector
Posted by Mark Thomas <ma...@apache.org>.
On 12/03/2013 21:47, Colin Ingarfield wrote:
> Hello,
>
> We're using Jetty 8.1.3.v20120416 w/ JDBCSessionManager as our application
> server with Tomcat 7's JDBC Connection pool 7.0.28. We've run this
> particular combination in production since at least Sept 2012. (Java 6 64
> bit/Ubuntu, Amazon RDS/MySQL db.)
>
> This morning one Jetty instance stopped accepting requests and the problem
> appears to be a deadlock btw a jetty connection handler thread and the conn
> pool's "PoolCleaner" thread. From jstack trace:
> Deadlock Detection:
>
> Found one Java-level deadlock:
> =============================
>
> "qtp1840392480-3740":
> waiting to lock Monitor@0x00007f4350001fd0 (Object@0x00000006c01a0e88, a
> com/mysql/jdbc/JDBC4Connection),
> which is held by "PoolCleaner[2009981184:1363034108768]"
> "PoolCleaner[2009981184:1363034108768]":
> waiting to lock Monitor@0x00007f4350001f28 (Object@0x00000006c1ed5738, a
> com/mysql/jdbc/JDBC4ResultSet),
> which is held by "qtp1840392480-3740"
>
> Once this happened all other worker threads blocked at various points
> within JDBCSession manager. Obviously restarting the instance resolved the
> issue for now.
>
> How does one go about diagnosing / resolving such an issue? It's not clear
> to me if either component is "at fault" here...I have the full jstack
> output as well as a full heap dump of the java process, if that would help.
Deadlocks occur when two threads attempt to acquire the same locks but
in a different order. Normally, one of the code paths is at fault. To
figure out which we need to see the full stack trace for the two threads
that are deadlocked.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org