[jira] [Created] (AMQ-6626) Deadlock caused by ActiveMQMessageConsumer.dispatch() due to onEvent invocation

["Facundo Velazquez (JIRA)" <ji...@apache.org>]

Facundo Velazquez created AMQ-6626:
--------------------------------------

             Summary: Deadlock caused by ActiveMQMessageConsumer.dispatch() due to onEvent invocation
                 Key: AMQ-6626
                 URL: https://issues.apache.org/jira/browse/AMQ-6626
             Project: ActiveMQ
          Issue Type: Bug
          Components: JMS client
            Reporter: Facundo Velazquez


Hi. 
In Mule ESB, 
I have detected a deadlock caused by the interaction between Spring and ActiveMQ.
Basically, there are two execution chains that are causing the deadlock:

First Chain: when a Message arrives to the JMS Queue:

1) ActiveMQMessageConsumer.dispatch() gets the lock of FifoMessageDispatchChannel.mutex.
2) ActiveMQMessageConsumer.dispatch() calls the onMessage() method without releasing the lock.
3) onMessage Mule Implementation calls org.springframework.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke() because It needs a toString of the used resource (in this case the connection) to logging purposes.
3) in ToString() method, org.springframework.SingleConnectionFactory.getConnection() method is called and It takes the org.springframework.SingleConnectionFactory.connectionMonitor lock to access to connection (that's a shared resource).

Lock ordering in this chain:

FifoMessageDispatchChannel.mutex lock -> org.springframework.SingleConnectionFactory.connectionMonitor lock

Second Chain: when A reconnection-strategy is applied:

1) Mule stops the current connection calling org.springframework.SingleConnectionFactory.localStop()
2) LocalStop takes the org.springframework.SingleConnectionFactory.connectionMonitor lock.
3) After a chain of calls: FifoMessageDispatchChannel.stop() is called.
4) FifoMessageDispatchChannel.stop() takes the FifoMessageDispatchChannel.mutex lock.

Lock Ordering in this case: 

org.springframework.SingleConnectionFactory.connectionMonitor lock -> FifoMessageDispatchChannel.mutex lock 

As you can see, The chains don't take care about lock ordering, and It's causing a deadlock.

I've sent a mail with same subject to the user list.
Although I agree the best and faster solution is to not use the string() method of the connection in log (because It's taking a lock), and I'll remove it in Mule code, I think would be desirable to not call onEvent method without realising the adquired lock. 

Thanks.




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)