ActiveMQ non-conformance to JMS Spec causing deadlock when using 3rd-party Resource Adapter

[rodos77 <eu...@nexj.com>]

The following is an excerpt from the JMS Spec 1.1:

8.2.3 ServerSessionPool

	...<snip>...

	If the ServerSessionPool is out of ServerSessions, the getServerSession()
method
	may block. If a ConnectionConsumer is blocked, it cannot deliver new
messages
	until a ServerSession is eventually returned.


I am using a 3rd-party JMS Resource Adapter to integrate ActiveMQ with a
J2EE app server.  This Resource Adapter does just what the JMS Spec above
allows - it blocks in the ServerSessionPool.getServerSession() call when it
is out of ServerSessions.  This causes ActiveMQ to hang due to a deadlock.

The deadlock occurs because the ServerSessionPool uses a single connection
to service all its ServerSessions.  This is perfectly legal.  However,
ActiveMQ is using this same connection for both dispatching messages to the
ConnectionConsumer as well as communicating transaction commits to the JMS
engine.  

So the deadlock scenario is as follows:
- message 1 is dispatched to the ConnectionConsumer and is being processed
- before message 1 is committed, message 2 is dispatched
- ServerSession is requested for processing message 2
- ServerSessionPool.getServerSession() blocks because it is out of
ServerSessions, blocking the connection from which the call was made
- message 1 is finished processing and the tx is committed
- ActiveMQ does a synchronous send using the same connection that is
currently blocked, to communicate the transaction commit command
- the commit blocks waiting for the connection to be released which will
never happen because that release is dependent on the ServerSession being
returned to the pool which won't happen until the tx is committed

The stacktraces of the 2 deadlocked threads at the time of the deadlock is
as follows:

"ActiveMQ Transport: tcp://localhost/127.0.0.1:61616" prio=6 tid=0x0aecc9f8
nid=0x1414 in Object.wait() [0x0b6af000..0x0b6afc68]
	at java.lang.Object.wait(Native Method)
	- waiting on <0x03070188> (a
org.apache.activemq.BlockingServerSessionPoolDeadlockTest$TestServerSessionPool)
	at java.lang.Object.wait(Object.java:474)
	at
org.apache.activemq.BlockingServerSessionPoolDeadlockTest$TestServerSessionPool.getServerSession(BlockingServerSessionPoolDeadlockTest.java:106)
	- locked <0x03070188> (a
org.apache.activemq.BlockingServerSessionPoolDeadlockTest$TestServerSessionPool)
	at
org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:129)
	at
org.apache.activemq.ActiveMQConnection$2.processMessageDispatch(ActiveMQConnection.java:1676)
	at
org.apache.activemq.command.MessageDispatch.visit(MessageDispatch.java:108)
	at
org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1658)
	at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:109)
	at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
	at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
	at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
	- locked <0x03064e98> (a org.apache.activemq.transport.InactivityMonitor$1)
	at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
	at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
	at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
	at java.lang.Thread.run(Thread.java:595)

	
"Thread-5" prio=6 tid=0x0af48ae0 nid=0x14a4 waiting on condition
[0x0b8af000..0x0b8afbe8]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
	at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:341)
	at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
	at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
	at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1244)
	at
org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:643)
	at
org.apache.activemq.TransactionContext.commit(TransactionContext.java:286)
	at org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:510)
	at
org.apache.activemq.BlockingServerSessionPoolDeadlockTest$TestServerSession$1.run(BlockingServerSessionPoolDeadlockTest.java:140)

Here's a junit test that illustrates this deadlock: 
http://old.nabble.com/file/p27869447/BlockingServerSessionPoolDeadlockTest.java
BlockingServerSessionPoolDeadlockTest.java 

Now, I realize that this bug does not manifest itself if the ActiveMQ
Resource Adapter is used.  However, that is only because of the way AMQ RA
works - it reuses existing ServerSessions instead of blocking when it runs
out (and does a weird callback from the jms session to the ServerSession to
start a tx, in order to avoid delivering several messages on same tx which
is what would happen when reusing an existing ServerSession that is already
in the middle of processing another message).  

I think this is a bug in AMQ JMS engine's implementation which just happens
to be masked by the way AMQ RA is implemented.  If a Resource Adapted relies
on the blocking behavior clearly provisioned by the JMS Spec, the problem
becomes evident.
-- 
View this message in context: http://old.nabble.com/ActiveMQ-non-conformance-to-JMS-Spec-causing-deadlock-when-using-3rd-party-Resource-Adapter-tp27869447p27869447.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ non-conformance to JMS Spec causing deadlock when using 3rd-party Resource Adapter

[rodos77 <eu...@nexj.com>]

JIRA AMQ-2652 created:  https://issues.apache.org/activemq/browse/AMQ-2652
https://issues.apache.org/activemq/browse/AMQ-2652 

-- 
View this message in context: http://old.nabble.com/ActiveMQ-non-conformance-to-JMS-Spec-causing-deadlock-when-using-3rd-party-Resource-Adapter-tp27869447p27910830.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.