You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Mats Henrikson (JIRA)" <ji...@apache.org> on 2009/07/28 00:29:33 UTC
[jira] Created: (AMQ-2332) it's possible to cause a deadlock in
ActiveMQ 5.2 using the client
it's possible to cause a deadlock in ActiveMQ 5.2 using the client
------------------------------------------------------------------
Key: AMQ-2332
URL: https://issues.apache.org/activemq/browse/AMQ-2332
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.2.0
Reporter: Mats Henrikson
Hi,
We have found a deadlock in ActiveMQ 5.2. It's possible that we may be causing it by a multi-threading problem in the client, we are currently looking into this. However, I feel that this is still a bug in the broker since it shouldn't be possible to deadlock the broker by incorrect use of the client.
The situation is that we have two clients connected to the broker, one over TCP using the Java library, and one connecting over SSL (our own implementation, this may also be involved in creating the problem) using the NMS library in .NET.
The Java client connects fine and subscribes to some topics and queues and starts sending messages to the topics. The NMS client connects and attempts to connect to some of the same topics, as well as creating a temporary queue. As it is doing this initial setup stuff it hangs and goes no further. From this point I don't think any more messages get delivered on the topic, and Java clients trying to send messages to it hang on the producers send() call indefinitely.
The deadlock appears to occur because a thread is stuck actually writing data back to the NMS client (the bottom stack trace below), another thread (originating from the Java client) is trying to acquire the write lock so that it too can write to the client (middle stack trace) while still owning a lock on the list of consumers of a topic, meanwhile a third thread (this one origination from the NMS client again) is trying to get that lock on the list of consumers in order to add to it (top stack trace). For some reason the first thread trying to write to the NMS client never finishes, even when the NMS client is killed, and so the lock is never released and everything involving that topic grinds to a halt.
I have looked but haven't been able to find anybody else reporting this issue.
I enabled JMX as well as debug mode and got the following stack traces from jconsole, and tracked down the locks involved in the deadlock using a debugger:
Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
State: BLOCKED on java.util.concurrent.CopyOnWriteArrayList@509c6c30 owned by: ActiveMQ Transport: tcp:///192.168.32.73:45666
Stack trace:
org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
java.lang.Thread.run(Thread.java:595)
CopyOnWriteArrayList@509c6c30 lock acquired at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49) below:
Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
State: BLOCKED on java.lang.Object@58c1c2f1 owned by: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
Stack trace:
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
org.apache.activemq.broker.region.Topic.send(Topic.java:365)
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
java.lang.Thread.run(Thread.java:595)
Object@58c1c2f1 lock acquired at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) below:
Name: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
State: RUNNABLE
Total blocked: 55 Total waited: 11
Stack trace:
java.net.SocketOutputStream.socketWrite0(Native Method)
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
java.io.DataOutputStream.flush(DataOutputStream.java:106)
org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
Let me know if I can do anything else to help.
--
Mats Henrikson
Software Engineer
SunGard Kiodex
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AMQ-2332) it's possible to cause a deadlock in
ActiveMQ 5.2 using the client
Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rob Davies reassigned AMQ-2332:
-------------------------------
Assignee: Rob Davies
> it's possible to cause a deadlock in ActiveMQ 5.2 using the client
> ------------------------------------------------------------------
>
> Key: AMQ-2332
> URL: https://issues.apache.org/activemq/browse/AMQ-2332
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Mats Henrikson
> Assignee: Rob Davies
>
> Hi,
> We have found a deadlock in ActiveMQ 5.2. It's possible that we may be causing it by a multi-threading problem in the client, we are currently looking into this. However, I feel that this is still a bug in the broker since it shouldn't be possible to deadlock the broker by incorrect use of the client.
> The situation is that we have two clients connected to the broker, one over TCP using the Java library, and one connecting over SSL (our own implementation, this may also be involved in creating the problem) using the NMS library in .NET.
> The Java client connects fine and subscribes to some topics and queues and starts sending messages to the topics. The NMS client connects and attempts to connect to some of the same topics, as well as creating a temporary queue. As it is doing this initial setup stuff it hangs and goes no further. From this point I don't think any more messages get delivered on the topic, and Java clients trying to send messages to it hang on the producers send() call indefinitely.
> The deadlock appears to occur because a thread is stuck actually writing data back to the NMS client (the bottom stack trace below), another thread (originating from the Java client) is trying to acquire the write lock so that it too can write to the client (middle stack trace) while still owning a lock on the list of consumers of a topic, meanwhile a third thread (this one origination from the NMS client again) is trying to get that lock on the list of consumers in order to add to it (top stack trace). For some reason the first thread trying to write to the NMS client never finishes, even when the NMS client is killed, and so the lock is never released and everything involving that topic grinds to a halt.
> I have looked but haven't been able to find anybody else reporting this issue.
> I enabled JMX as well as debug mode and got the following stack traces from jconsole, and tracked down the locks involved in the deadlock using a debugger:
> Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
> State: BLOCKED on java.util.concurrent.CopyOnWriteArrayList@509c6c30 owned by: ActiveMQ Transport: tcp:///192.168.32.73:45666
> Stack trace:
> org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
> org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> CopyOnWriteArrayList@509c6c30 lock acquired at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49) below:
> Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
> State: BLOCKED on java.lang.Object@58c1c2f1 owned by: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> Stack trace:
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
> org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
> org.apache.activemq.broker.region.Topic.send(Topic.java:365)
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> Object@58c1c2f1 lock acquired at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) below:
> Name: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> State: RUNNABLE
> Total blocked: 55 Total waited: 11
> Stack trace:
> java.net.SocketOutputStream.socketWrite0(Native Method)
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
> com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
> com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> java.io.DataOutputStream.flush(DataOutputStream.java:106)
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
> org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
> org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
> org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Let me know if I can do anything else to help.
> --
> Mats Henrikson
> Software Engineer
> SunGard Kiodex
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-2332) it's possible to cause a deadlock in
ActiveMQ 5.2 using the client
Posted by "Rob Davies (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rob Davies resolved AMQ-2332.
-----------------------------
Fix Version/s: 5.4.0
(was: 5.4.1)
Resolution: Fixed
Socket timeouts are OS specific - but you can set the property on the transport used by the broker - see http://activemq.apache.org/tcp-transport-reference.html
> it's possible to cause a deadlock in ActiveMQ 5.2 using the client
> ------------------------------------------------------------------
>
> Key: AMQ-2332
> URL: https://issues.apache.org/activemq/browse/AMQ-2332
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Mats Henrikson
> Assignee: Rob Davies
> Fix For: 5.4.0
>
>
> Hi,
> We have found a deadlock in ActiveMQ 5.2. It's possible that we may be causing it by a multi-threading problem in the client, we are currently looking into this. However, I feel that this is still a bug in the broker since it shouldn't be possible to deadlock the broker by incorrect use of the client.
> The situation is that we have two clients connected to the broker, one over TCP using the Java library, and one connecting over SSL (our own implementation, this may also be involved in creating the problem) using the NMS library in .NET.
> The Java client connects fine and subscribes to some topics and queues and starts sending messages to the topics. The NMS client connects and attempts to connect to some of the same topics, as well as creating a temporary queue. As it is doing this initial setup stuff it hangs and goes no further. From this point I don't think any more messages get delivered on the topic, and Java clients trying to send messages to it hang on the producers send() call indefinitely.
> The deadlock appears to occur because a thread is stuck actually writing data back to the NMS client (the bottom stack trace below), another thread (originating from the Java client) is trying to acquire the write lock so that it too can write to the client (middle stack trace) while still owning a lock on the list of consumers of a topic, meanwhile a third thread (this one origination from the NMS client again) is trying to get that lock on the list of consumers in order to add to it (top stack trace). For some reason the first thread trying to write to the NMS client never finishes, even when the NMS client is killed, and so the lock is never released and everything involving that topic grinds to a halt.
> I have looked but haven't been able to find anybody else reporting this issue.
> I enabled JMX as well as debug mode and got the following stack traces from jconsole, and tracked down the locks involved in the deadlock using a debugger:
> Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
> State: BLOCKED on java.util.concurrent.CopyOnWriteArrayList@509c6c30 owned by: ActiveMQ Transport: tcp:///192.168.32.73:45666
> Stack trace:
> org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
> org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> CopyOnWriteArrayList@509c6c30 lock acquired at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49) below:
> Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
> State: BLOCKED on java.lang.Object@58c1c2f1 owned by: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> Stack trace:
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
> org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
> org.apache.activemq.broker.region.Topic.send(Topic.java:365)
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> Object@58c1c2f1 lock acquired at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) below:
> Name: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> State: RUNNABLE
> Total blocked: 55 Total waited: 11
> Stack trace:
> java.net.SocketOutputStream.socketWrite0(Native Method)
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
> com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
> com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> java.io.DataOutputStream.flush(DataOutputStream.java:106)
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
> org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
> org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
> org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Let me know if I can do anything else to help.
> --
> Mats Henrikson
> Software Engineer
> SunGard Kiodex
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2332) it's possible to cause a deadlock in
ActiveMQ 5.2 using the client
Posted by "Mats Henrikson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=53444#action_53444 ]
Mats Henrikson commented on AMQ-2332:
-------------------------------------
I believe that the most likely cause was a non-optimal SSL implementation we tried to write for the NMS C# client; we have since stopped using the SSL implementation and now use IKVM to run the Java client on .NET and the deadlocks have stopped.
I have not seen the SSL implementation we did myself, but looking at the stack trace it looks like it might be possible to recreate it by not reading the complete SSL response. I don't know enough about how Java and C# does network communication to say why the socket never timed out, even after the C# client was killed.
> it's possible to cause a deadlock in ActiveMQ 5.2 using the client
> ------------------------------------------------------------------
>
> Key: AMQ-2332
> URL: https://issues.apache.org/activemq/browse/AMQ-2332
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Mats Henrikson
> Assignee: Rob Davies
>
> Hi,
> We have found a deadlock in ActiveMQ 5.2. It's possible that we may be causing it by a multi-threading problem in the client, we are currently looking into this. However, I feel that this is still a bug in the broker since it shouldn't be possible to deadlock the broker by incorrect use of the client.
> The situation is that we have two clients connected to the broker, one over TCP using the Java library, and one connecting over SSL (our own implementation, this may also be involved in creating the problem) using the NMS library in .NET.
> The Java client connects fine and subscribes to some topics and queues and starts sending messages to the topics. The NMS client connects and attempts to connect to some of the same topics, as well as creating a temporary queue. As it is doing this initial setup stuff it hangs and goes no further. From this point I don't think any more messages get delivered on the topic, and Java clients trying to send messages to it hang on the producers send() call indefinitely.
> The deadlock appears to occur because a thread is stuck actually writing data back to the NMS client (the bottom stack trace below), another thread (originating from the Java client) is trying to acquire the write lock so that it too can write to the client (middle stack trace) while still owning a lock on the list of consumers of a topic, meanwhile a third thread (this one origination from the NMS client again) is trying to get that lock on the list of consumers in order to add to it (top stack trace). For some reason the first thread trying to write to the NMS client never finishes, even when the NMS client is killed, and so the lock is never released and everything involving that topic grinds to a halt.
> I have looked but haven't been able to find anybody else reporting this issue.
> I enabled JMX as well as debug mode and got the following stack traces from jconsole, and tracked down the locks involved in the deadlock using a debugger:
> Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
> State: BLOCKED on java.util.concurrent.CopyOnWriteArrayList@509c6c30 owned by: ActiveMQ Transport: tcp:///192.168.32.73:45666
> Stack trace:
> org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
> org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> CopyOnWriteArrayList@509c6c30 lock acquired at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49) below:
> Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
> State: BLOCKED on java.lang.Object@58c1c2f1 owned by: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> Stack trace:
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
> org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
> org.apache.activemq.broker.region.Topic.send(Topic.java:365)
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> Object@58c1c2f1 lock acquired at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) below:
> Name: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> State: RUNNABLE
> Total blocked: 55 Total waited: 11
> Stack trace:
> java.net.SocketOutputStream.socketWrite0(Native Method)
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
> com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
> com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> java.io.DataOutputStream.flush(DataOutputStream.java:106)
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
> org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
> org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
> org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Let me know if I can do anything else to help.
> --
> Mats Henrikson
> Software Engineer
> SunGard Kiodex
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2332) it's possible to cause a deadlock in
ActiveMQ 5.2 using the client
Posted by "Mats Henrikson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=52981#action_52981 ]
Mats Henrikson commented on AMQ-2332:
-------------------------------------
Versions for the broker:
ActiveMQ 5.2 release
java version "1.5.0_14"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_14-b03, mixed mode)
The Java client is using the jar out of the broker package.
The broker and Java client are running on some RedHat Linux 64 bit.
The NMS client is version 1.2 I believe, and runs on Windows XP.
> it's possible to cause a deadlock in ActiveMQ 5.2 using the client
> ------------------------------------------------------------------
>
> Key: AMQ-2332
> URL: https://issues.apache.org/activemq/browse/AMQ-2332
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Mats Henrikson
>
> Hi,
> We have found a deadlock in ActiveMQ 5.2. It's possible that we may be causing it by a multi-threading problem in the client, we are currently looking into this. However, I feel that this is still a bug in the broker since it shouldn't be possible to deadlock the broker by incorrect use of the client.
> The situation is that we have two clients connected to the broker, one over TCP using the Java library, and one connecting over SSL (our own implementation, this may also be involved in creating the problem) using the NMS library in .NET.
> The Java client connects fine and subscribes to some topics and queues and starts sending messages to the topics. The NMS client connects and attempts to connect to some of the same topics, as well as creating a temporary queue. As it is doing this initial setup stuff it hangs and goes no further. From this point I don't think any more messages get delivered on the topic, and Java clients trying to send messages to it hang on the producers send() call indefinitely.
> The deadlock appears to occur because a thread is stuck actually writing data back to the NMS client (the bottom stack trace below), another thread (originating from the Java client) is trying to acquire the write lock so that it too can write to the client (middle stack trace) while still owning a lock on the list of consumers of a topic, meanwhile a third thread (this one origination from the NMS client again) is trying to get that lock on the list of consumers in order to add to it (top stack trace). For some reason the first thread trying to write to the NMS client never finishes, even when the NMS client is killed, and so the lock is never released and everything involving that topic grinds to a halt.
> I have looked but haven't been able to find anybody else reporting this issue.
> I enabled JMX as well as debug mode and got the following stack traces from jconsole, and tracked down the locks involved in the deadlock using a debugger:
> Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
> State: BLOCKED on java.util.concurrent.CopyOnWriteArrayList@509c6c30 owned by: ActiveMQ Transport: tcp:///192.168.32.73:45666
> Stack trace:
> org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
> org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> CopyOnWriteArrayList@509c6c30 lock acquired at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49) below:
> Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
> State: BLOCKED on java.lang.Object@58c1c2f1 owned by: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> Stack trace:
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
> org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
> org.apache.activemq.broker.region.Topic.send(Topic.java:365)
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> Object@58c1c2f1 lock acquired at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) below:
> Name: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> State: RUNNABLE
> Total blocked: 55 Total waited: 11
> Stack trace:
> java.net.SocketOutputStream.socketWrite0(Native Method)
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
> com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
> com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> java.io.DataOutputStream.flush(DataOutputStream.java:106)
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
> org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
> org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
> org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Let me know if I can do anything else to help.
> --
> Mats Henrikson
> Software Engineer
> SunGard Kiodex
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-2332) it's possible to cause a deadlock in
ActiveMQ 5.2 using the client
Posted by "Mats Henrikson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-2332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=60594#action_60594 ]
Mats Henrikson commented on AMQ-2332:
-------------------------------------
This has been resolved as Fixed, was it really? Because the last comment on the ticket reads to me like it was just closed saying I should adjust my socket timeout.
I also don't think this has anything to do with socket timeouts - as long as the client stays connected the deadlock will persist (and it may even persist after the client has disconnected...)
Anyway, just wondering if it was "Fixed" or if it should be marked "Wont do"...
> it's possible to cause a deadlock in ActiveMQ 5.2 using the client
> ------------------------------------------------------------------
>
> Key: AMQ-2332
> URL: https://issues.apache.org/activemq/browse/AMQ-2332
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.2.0
> Reporter: Mats Henrikson
> Assignee: Rob Davies
> Fix For: 5.4.0
>
>
> Hi,
> We have found a deadlock in ActiveMQ 5.2. It's possible that we may be causing it by a multi-threading problem in the client, we are currently looking into this. However, I feel that this is still a bug in the broker since it shouldn't be possible to deadlock the broker by incorrect use of the client.
> The situation is that we have two clients connected to the broker, one over TCP using the Java library, and one connecting over SSL (our own implementation, this may also be involved in creating the problem) using the NMS library in .NET.
> The Java client connects fine and subscribes to some topics and queues and starts sending messages to the topics. The NMS client connects and attempts to connect to some of the same topics, as well as creating a temporary queue. As it is doing this initial setup stuff it hangs and goes no further. From this point I don't think any more messages get delivered on the topic, and Java clients trying to send messages to it hang on the producers send() call indefinitely.
> The deadlock appears to occur because a thread is stuck actually writing data back to the NMS client (the bottom stack trace below), another thread (originating from the Java client) is trying to acquire the write lock so that it too can write to the client (middle stack trace) while still owning a lock on the list of consumers of a topic, meanwhile a third thread (this one origination from the NMS client again) is trying to get that lock on the list of consumers in order to add to it (top stack trace). For some reason the first thread trying to write to the NMS client never finishes, even when the NMS client is killed, and so the lock is never released and everything involving that topic grinds to a halt.
> I have looked but haven't been able to find anybody else reporting this issue.
> I enabled JMX as well as debug mode and got the following stack traces from jconsole, and tracked down the locks involved in the deadlock using a debugger:
> Name: ActiveMQ Transport: ssl:///10.247.39.150:59759
> State: BLOCKED on java.util.concurrent.CopyOnWriteArrayList@509c6c30 owned by: ActiveMQ Transport: tcp:///192.168.32.73:45666
> Stack trace:
> org.apache.activemq.broker.region.Topic.addSubscription(Topic.java:147)
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:275)
> org.apache.activemq.broker.region.TopicRegion.addConsumer(TopicRegion.java:108)
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:375)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:83)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:86)
> org.apache.activemq.security.AuthorizationBroker.addConsumer(AuthorizationBroker.java:138)
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:93)
> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:541)
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:345)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.SslTransport.doConsume(SslTransport.java:104)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> CopyOnWriteArrayList@509c6c30 lock acquired at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49) below:
> Name: ActiveMQ Transport: tcp:///192.168.32.73:45666
> State: BLOCKED on java.lang.Object@58c1c2f1 owned by: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> Stack trace:
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:39)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.TopicSubscription.dispatch(TopicSubscription.java:427)
> org.apache.activemq.broker.region.TopicSubscription.add(TopicSubscription.java:90)
> org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
> org.apache.activemq.broker.region.Topic.dispatch(Topic.java:580)
> org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:427)
> org.apache.activemq.broker.region.Topic.send(Topic.java:365)
> org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:350)
> org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:437)
> org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:126)
> org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:184)
> org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
> org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
> org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> java.lang.Thread.run(Thread.java:595)
> Object@58c1c2f1 lock acquired at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) below:
> Name: TempQueue: c34c73c1-d2e9-41f3-89a6-82f2c59ca62b:1
> State: RUNNABLE
> Total blocked: 55 Total waited: 11
> Stack trace:
> java.net.SocketOutputStream.socketWrite0(Native Method)
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> com.sun.net.ssl.internal.ssl.OutputRecord.writeBuffer(OutputRecord.java:295)
> com.sun.net.ssl.internal.ssl.OutputRecord.write(OutputRecord.java:284)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:727)
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:715)
> com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
> org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
> java.io.DataOutputStream.flush(DataOutputStream.java:106)
> org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:167)
> org.apache.activemq.transport.InactivityMonitor.oneway(InactivityMonitor.java:233)
> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:83)
> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:100)
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1207)
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:793)
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:752)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:640)
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:588)
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:150)
> org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1266)
> org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1223)
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1308)
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1011)
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
> Let me know if I can do anything else to help.
> --
> Mats Henrikson
> Software Engineer
> SunGard Kiodex
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.