You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Sam hendley (JIRA)" <ji...@apache.org> on 2015/10/22 15:54:27 UTC
[jira] [Commented] (AMQ-4361) Deadlock during close while
publishing to flow-controlled queue
[ https://issues.apache.org/jira/browse/AMQ-4361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969183#comment-14969183 ]
Sam hendley commented on AMQ-4361:
----------------------------------
We recently updated to 5.12.0 and a test I had put in place to track the "higher level" shutdown started failing occasionally. I still haven't worked out the cause of that but I did try enabling ProducerWindowSize to verify the fix to this bug. So technically it closes quickly but it is throwing an IllegalMonitorStateException. There is a bug in the waitForSpace method where we don't relock the readLock if the waitForSpaceCondition throws.
{{
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread
at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:447)
at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:431)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1340)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:883)
at org.apache.activemq.usage.MemoryUsage.waitForSpace(MemoryUsage.java:87)
at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:282)
at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:223)
at org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:269)
at org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:181)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:636)
at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:607)
at org.springframework.jms.core.JmsTemplate$3.doInJms(JmsTemplate.java:572)
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:494)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:569)
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:560)
at com.sms.jms.FastInterruptibleSingleDestinationJmsOutClient.send(FastInterruptibleSingleDestinationJmsOutClient.java:77)
at com.sms.jms.FastInterruptibleSingleDestinationJmsOutClient.publishAsBytesMessage(FastInterruptibleSingleDestinationJmsOutClient.java:68)
at com.sms.jms.FastInterruptibleSingleDestinationJmsOutClientTest$2.run(FastInterruptibleSingleDestinationJmsOutClientTest.java:83)
at java.lang.Thread.run(Thread.java:744)
}}
{code}
public boolean waitForSpace(long timeout) throws InterruptedException {
if (parent != null) {
if (!parent.waitForSpace(timeout)) {
return false;
}
}
usageLock.readLock().lock();
try {
if (percentUsage >= 100) {
usageLock.readLock().unlock();
usageLock.writeLock().lock();
try {
while (percentUsage >= 100 ) {
waitForSpaceCondition.await(timeout, TimeUnit.MILLISECONDS);
}
// I believe this relocking needs to be in the finally block.
usageLock.readLock().lock();
} finally {
usageLock.writeLock().unlock();
}
}
return percentUsage < 100;
} finally {
usageLock.readLock().unlock();
}
}
{code}
> Deadlock during close while publishing to flow-controlled queue
> ---------------------------------------------------------------
>
> Key: AMQ-4361
> URL: https://issues.apache.org/jira/browse/AMQ-4361
> Project: ActiveMQ
> Issue Type: Bug
> Components: JMS client
> Affects Versions: 5.6.0, 5.8.0
> Environment: Windows, Linux, JDK 1.6
> Reporter: Sam hendley
> Assignee: Timothy Bish
> Fix For: 5.9.0
>
>
> TestCase on github:
> https://github.com/samhendley/activemq-close-deadlock-bug
> The deadlock occurs when we are using TcpTransport to a flow-controlled queue and we then try to gracefully shutdown the application. The close operation hangs forever because it is trying to send a "close packet" to the server. It never gets the chance to send that request because the socket is blocked by the publishing thread. This stops my publisher from shutting down and causes us to orphan threads during shutdown.
> I have verified this bug occurs on atleast activemq 5.6.0 and 5.8.0 and on linux and windows using JDK 1.6.
> I don't need a fix for the bug necessarily, just a way to gracefully shutdown my publisher if I get into this state.
> Partial Stack Trace During failure
> "ClosingThread" prio=6 tid=0x045ce000 nid=0xa84 waiting on condition [0x04ddf000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x23fc52d8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
> at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
> at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
> at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:66)
> at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1304)
> at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1298)
> at org.apache.activemq.ActiveMQSession.asyncSendPacket(ActiveMQSession.java:1901)
> at org.apache.activemq.ActiveMQMessageProducer.close(ActiveMQMessageProducer.java:173)
> at org.activemq.bug.DeadlockDuringCloseTest$2.run(DeadlockDuringCloseTest.java:83)
> at java.lang.Thread.run(Thread.java:662)
> "PublishingThread" prio=6 tid=0x045cd800 nid=0xb84 runnable [0x04d8f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketOutputStream.socketWrite0(Native Method)
> at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> at java.io.DataOutputStream.flush(DataOutputStream.java:106)
> at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:176)
> at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:322)
> at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:304)
> at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
> at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
> at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
> at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1304)
> at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1298)
> at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1782)
> - locked <0x23faa7d8> (a java.lang.Object)
> at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:289)
> at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:224)
> at org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:300)
> at org.activemq.bug.DeadlockDuringCloseTest$1.run(DeadlockDuringCloseTest.java:63)
> at java.lang.Thread.run(Thread.java:662)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)