You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by drvillo <f....@gmail.com> on 2006/12/15 17:41:11 UTC
Random deadlocks or lost messages with AMQ 4.1 and Jencks
Hi there
I have spent the whole day now tracking down this, and I have no clue.
I have stripped everything down to a silly system:
sender1->queue1->listener1->sender2->queue2->listener2
I have a test that basically sends 10 messages with sender1, no more.
The listeners are just logging the incoming messages, and listener1 has
a sender himself, to resend the same message to the second queue.
Actually the situation is the following
-sometimes I don't manage to send all the messages, because of a deadlock
(given below)
-sometimes I get no deadlocks but I loose a message (always the second) (and
from
the jconsole I can see that there's a message stuck in the queue and not
consumed)
-sometimes it goes fine
the deadlock is reported below, with the last logging information appeared
DEBUG org.apache.activemq.broker.region.AbstractRegion - Removing consumer:
ID:obimac.local-51149-1166199529487-2:2:-1:4
DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - run loop end
DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - Session returned to
pool: ServerSessionImpl:1
DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Run finished
DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Work completed:
javax.resource.spi.work.WorkEvent[source=Work :ServerSessionImpl:1]
DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - ServerSession
requested.
...
Found one Java-level deadlock:
=============================
"ActiveMQ Task":
waiting to lock monitor 0x018123fc (object 0x26b8fa00, a
java.lang.Object),
which is held by "Thread-3"
"Thread-3":
waiting to lock monitor 0x018123d8 (object 0x26b8e848, a
java.lang.Object),
which is held by "ActiveMQ Task"
Java stack information for the threads listed above:
===================================================
"ActiveMQ Task":
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
- waiting to lock <0x26b8fa00> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1151)
at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:226)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:277)
at
org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl(ServerSessionPoolImpl.java:63)
at
org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:116)
at
org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:136)
at
org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1473)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at
org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
- locked <0x26b8e848> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
at
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:772)
at
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:788)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:111)
at
org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:613)
"Thread-3":
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
- waiting to lock <0x26b8e848> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at
org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
- locked <0x26b8fa00> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1171)
at
org.apache.activemq.TransactionContext.commit(TransactionContext.java:260)
at
org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:58)
at
org.jencks.LocalTransactionEndpoint.afterDelivery(LocalTransactionEndpoint.java:52)
at
org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
at
org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
at
org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:216)
at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
at
org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:613)
Found 1 deadlock.
Attached there's my spring config file. I have been trying switching
async dispatches and sends off, thinking that could have been the problem
but that
didn't help. turning off message persistence doesn't change anything either.
Needless to say that the whole of this is pretty frustrating, in the last
couple of days I have been
dealing with deadlocks all over the place.
I'm running AMC 4.1, jencks 1.3 spring 2.0.
I hope someone will help on this, otherwise I'll probably have to drop the
whole project:(
thanks
Francesco Vivoli
http://www.nabble.com/file/4735/container.xml container.xml
--
View this message in context: http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7894874
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Random deadlocks or lost messages with AMQ 4.1 and Jencks
Posted by drvillo <f....@gmail.com>.
Is there any estimate of how long soon means?
It would help for planning...
BTW, I guess that even if I don't use jboss it's the same right?
my concerns are whether my configuration is sound or not.
Is it so?
Thanks for the quick reply anyway
Francesco
James.Strachan wrote:
>
> This is a known issue...
>
> http://issues.apache.org/activemq/browse/AMQ-1092
>
> we'll hopefully have it fixed soon
>
> On 12/15/06, drvillo <f....@gmail.com> wrote:
>>
>> Hi there
>>
>> I have spent the whole day now tracking down this, and I have no clue.
>> I have stripped everything down to a silly system:
>> sender1->queue1->listener1->sender2->queue2->listener2
>>
>>
>> I have a test that basically sends 10 messages with sender1, no more.
>>
>> The listeners are just logging the incoming messages, and listener1 has
>> a sender himself, to resend the same message to the second queue.
>>
>>
>> Actually the situation is the following
>> -sometimes I don't manage to send all the messages, because of a deadlock
>> (given below)
>> -sometimes I get no deadlocks but I loose a message (always the second)
>> (and
>> from
>> the jconsole I can see that there's a message stuck in the queue and not
>> consumed)
>> -sometimes it goes fine
>>
>>
>> the deadlock is reported below, with the last logging information
>> appeared
>>
>> DEBUG org.apache.activemq.broker.region.AbstractRegion - Removing
>> consumer:
>> ID:obimac.local-51149-1166199529487-2:2:-1:4
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - run loop end
>> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - Session returned to
>> pool: ServerSessionImpl:1
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Run finished
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Work completed:
>> javax.resource.spi.work.WorkEvent[source=Work :ServerSessionImpl:1]
>> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - ServerSession
>> requested.
>>
>>
>> ...
>>
>> Found one Java-level deadlock:
>> =============================
>> "ActiveMQ Task":
>> waiting to lock monitor 0x018123fc (object 0x26b8fa00, a
>> java.lang.Object),
>> which is held by "Thread-3"
>> "Thread-3":
>> waiting to lock monitor 0x018123d8 (object 0x26b8e848, a
>> java.lang.Object),
>> which is held by "ActiveMQ Task"
>>
>> Java stack information for the threads listed above:
>> ===================================================
>> "ActiveMQ Task":
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
>> - waiting to lock <0x26b8fa00> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> at
>> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1151)
>> at
>> org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:226)
>> at
>> org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:277)
>> at
>> org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl(ServerSessionPoolImpl.java:63)
>> at
>> org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:116)
>> at
>> org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:136)
>> at
>> org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1473)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
>> at
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
>> at
>> org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
>> at
>> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
>> - locked <0x26b8e848> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> at
>> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
>> at
>> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:772)
>> at
>> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:788)
>> at
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:111)
>> at
>> org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
>> at
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44)
>> at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>> at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>> at java.lang.Thread.run(Thread.java:613)
>> "Thread-3":
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
>> - waiting to lock <0x26b8e848> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> at
>> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
>> at
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
>> at
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
>> at
>> org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
>> at
>> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
>> - locked <0x26b8fa00> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74)
>> at
>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1171)
>> at
>> org.apache.activemq.TransactionContext.commit(TransactionContext.java:260)
>> at
>> org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:58)
>> at
>> org.jencks.LocalTransactionEndpoint.afterDelivery(LocalTransactionEndpoint.java:52)
>> at
>> org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
>> at
>> org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
>> at
>> org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:216)
>> at
>> org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
>> at
>> org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
>> at
>> org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
>> at
>> EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
>> Source)
>> at java.lang.Thread.run(Thread.java:613)
>>
>> Found 1 deadlock.
>>
>>
>> Attached there's my spring config file. I have been trying switching
>> async dispatches and sends off, thinking that could have been the problem
>> but that
>> didn't help. turning off message persistence doesn't change anything
>> either.
>>
>>
>> Needless to say that the whole of this is pretty frustrating, in the last
>> couple of days I have been
>> dealing with deadlocks all over the place.
>> I'm running AMC 4.1, jencks 1.3 spring 2.0.
>>
>> I hope someone will help on this, otherwise I'll probably have to drop
>> the
>> whole project:(
>>
>> thanks
>> Francesco Vivoli
>>
>> http://www.nabble.com/file/4735/container.xml container.xml
>> --
>> View this message in context:
>> http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7894874
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7895176
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Random deadlocks or lost messages with AMQ 4.1 and Jencks
Posted by James Strachan <ja...@gmail.com>.
This is a known issue...
http://issues.apache.org/activemq/browse/AMQ-1092
we'll hopefully have it fixed soon
On 12/15/06, drvillo <f....@gmail.com> wrote:
>
> Hi there
>
> I have spent the whole day now tracking down this, and I have no clue.
> I have stripped everything down to a silly system:
> sender1->queue1->listener1->sender2->queue2->listener2
>
>
> I have a test that basically sends 10 messages with sender1, no more.
>
> The listeners are just logging the incoming messages, and listener1 has
> a sender himself, to resend the same message to the second queue.
>
>
> Actually the situation is the following
> -sometimes I don't manage to send all the messages, because of a deadlock
> (given below)
> -sometimes I get no deadlocks but I loose a message (always the second) (and
> from
> the jconsole I can see that there's a message stuck in the queue and not
> consumed)
> -sometimes it goes fine
>
>
> the deadlock is reported below, with the last logging information appeared
>
> DEBUG org.apache.activemq.broker.region.AbstractRegion - Removing consumer:
> ID:obimac.local-51149-1166199529487-2:2:-1:4
> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - run loop end
> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - Session returned to
> pool: ServerSessionImpl:1
> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Run finished
> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Work completed:
> javax.resource.spi.work.WorkEvent[source=Work :ServerSessionImpl:1]
> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - ServerSession
> requested.
>
>
> ...
>
> Found one Java-level deadlock:
> =============================
> "ActiveMQ Task":
> waiting to lock monitor 0x018123fc (object 0x26b8fa00, a
> java.lang.Object),
> which is held by "Thread-3"
> "Thread-3":
> waiting to lock monitor 0x018123d8 (object 0x26b8e848, a
> java.lang.Object),
> which is held by "ActiveMQ Task"
>
> Java stack information for the threads listed above:
> ===================================================
> "ActiveMQ Task":
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
> - waiting to lock <0x26b8fa00> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1151)
> at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:226)
> at
> org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:277)
> at
> org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl(ServerSessionPoolImpl.java:63)
> at
> org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:116)
> at
> org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:136)
> at
> org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1473)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
> at
> org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
> at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
> - locked <0x26b8e848> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
> at
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:772)
> at
> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:788)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:111)
> at
> org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44)
> at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:613)
> "Thread-3":
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
> - waiting to lock <0x26b8e848> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
> at
> org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
> at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
> - locked <0x26b8fa00> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1171)
> at
> org.apache.activemq.TransactionContext.commit(TransactionContext.java:260)
> at
> org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:58)
> at
> org.jencks.LocalTransactionEndpoint.afterDelivery(LocalTransactionEndpoint.java:52)
> at
> org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
> at
> org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
> at
> org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:216)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at
> org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:613)
>
> Found 1 deadlock.
>
>
> Attached there's my spring config file. I have been trying switching
> async dispatches and sends off, thinking that could have been the problem
> but that
> didn't help. turning off message persistence doesn't change anything either.
>
>
> Needless to say that the whole of this is pretty frustrating, in the last
> couple of days I have been
> dealing with deadlocks all over the place.
> I'm running AMC 4.1, jencks 1.3 spring 2.0.
>
> I hope someone will help on this, otherwise I'll probably have to drop the
> whole project:(
>
> thanks
> Francesco Vivoli
>
> http://www.nabble.com/file/4735/container.xml container.xml
> --
> View this message in context: http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7894874
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Random deadlocks or lost messages with AMQ 4.1 and Jencks
Posted by drvillo <f....@gmail.com>.
Hi
actually running with async turned on,
<property name="serverUrl"
value="vm://brokerConfigTest?jms.useAsyncSend=true&jms.dispatchAsync=true"/>
allows the test to run for 10 messages, but results in another deadlock to
be found
when trying with 1000 messages...
the above URI should be fine right? it is given to the resource adapter.
Thanks again, I'm trying now with tcp instead of the vm transport...
Found one Java-level deadlock:
=============================
"Thread-6":
waiting to lock monitor 0x018123d8 (object 0x26b8f0a0, a
java.lang.Object),
which is held by "ActiveMQ Task"
"ActiveMQ Task":
waiting to lock monitor 0x018123b4 (object 0x26b8e7b0, a
java.lang.Object),
which is held by "Thread-6"
Java stack information for the threads listed above:
===================================================
"Thread-6":
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:42)
- waiting to lock <0x26b8f0a0> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at
org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
- locked <0x26b8e7b0> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1171)
at
org.apache.activemq.TransactionContext.commit(TransactionContext.java:260)
at
org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:58)
at
org.jencks.LocalTransactionEndpoint.afterDelivery(LocalTransactionEndpoint.java:52)
at
org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
at
org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
at
org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:216)
at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
at
org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:613)
"ActiveMQ Task":
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:42)
- waiting to lock <0x26b8e7b0> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1151)
at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:226)
at
org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:277)
at
org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl(ServerSessionPoolImpl.java:63)
at
org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:116)
at
org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:136)
at
org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1473)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
at
org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
at
org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
- locked <0x26b8f0a0> (a java.lang.Object)
at
org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
at
org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
at
org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:772)
at
org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:788)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:111)
at
org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:613)
Found 1 deadlock.
rajdavies wrote:
>
> As you're using the vm://transport - you could set the async flag =
> true - which will cure this problem - e.g. your connection url would
> be vm://localhost?async=true
>
> cheers,
>
> Rob
> On 15 Dec 2006, at 16:41, drvillo wrote:
>
>>
>> Hi there
>>
>> I have spent the whole day now tracking down this, and I have no clue.
>> I have stripped everything down to a silly system:
>> sender1->queue1->listener1->sender2->queue2->listener2
>>
>>
>> I have a test that basically sends 10 messages with sender1, no more.
>>
>> The listeners are just logging the incoming messages, and listener1
>> has
>> a sender himself, to resend the same message to the second queue.
>>
>>
>> Actually the situation is the following
>> -sometimes I don't manage to send all the messages, because of a
>> deadlock
>> (given below)
>> -sometimes I get no deadlocks but I loose a message (always the
>> second) (and
>> from
>> the jconsole I can see that there's a message stuck in the queue
>> and not
>> consumed)
>> -sometimes it goes fine
>>
>>
>> the deadlock is reported below, with the last logging information
>> appeared
>>
>> DEBUG org.apache.activemq.broker.region.AbstractRegion - Removing
>> consumer:
>> ID:obimac.local-51149-1166199529487-2:2:-1:4
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - run loop end
>> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - Session
>> returned to
>> pool: ServerSessionImpl:1
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Run finished
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Work completed:
>> javax.resource.spi.work.WorkEvent[source=Work :ServerSessionImpl:1]
>> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - ServerSession
>> requested.
>>
>>
>> ...
>>
>> Found one Java-level deadlock:
>> =============================
>> "ActiveMQ Task":
>> waiting to lock monitor 0x018123fc (object 0x26b8fa00, a
>> java.lang.Object),
>> which is held by "Thread-3"
>> "Thread-3":
>> waiting to lock monitor 0x018123d8 (object 0x26b8e848, a
>> java.lang.Object),
>> which is held by "ActiveMQ Task"
>>
>> Java stack information for the threads listed above:
>> ===================================================
>> "ActiveMQ Task":
>> at
>> org.apache.activemq.transport.MutexTransport.oneway
>> (MutexTransport.java:43)
>> - waiting to lock <0x26b8fa00> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.oneway
>> (ResponseCorrelator.java:60)
>> at
>> org.apache.activemq.ActiveMQConnection.asyncSendPacket
>> (ActiveMQConnection.java:1151)
>> at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:
>> 226)
>> at
>> org.apache.activemq.ActiveMQConnection.createSession
>> (ActiveMQConnection.java:277)
>> at
>> org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl
>> (ServerSessionPoolImpl.java:63)
>> at
>> org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession
>> (ServerSessionPoolImpl.java:116)
>> at
>> org.apache.activemq.ActiveMQConnectionConsumer.dispatch
>> (ActiveMQConnectionConsumer.java:136)
>> at
>> org.apache.activemq.ActiveMQConnection.onCommand
>> (ActiveMQConnection.java:1473)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.onCommand
>> (ResponseCorrelator.java:95)
>> at
>> org.apache.activemq.transport.TransportFilter.onCommand
>> (TransportFilter.java:65)
>> at
>> org.apache.activemq.transport.vm.VMTransport.syncOneWay
>> (VMTransport.java:96)
>> at org.apache.activemq.transport.vm.VMTransport.oneway
>> (VMTransport.java:83)
>> at
>> org.apache.activemq.transport.MutexTransport.oneway
>> (MutexTransport.java:43)
>> - locked <0x26b8e848> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.oneway
>> (ResponseCorrelator.java:60)
>> at
>> org.apache.activemq.broker.TransportConnection.dispatch
>> (TransportConnection.java:1051)
>> at
>> org.apache.activemq.broker.TransportConnection.processDispatch
>> (TransportConnection.java:772)
>> at
>> org.apache.activemq.broker.TransportConnection.iterate
>> (TransportConnection.java:788)
>> at
>> org.apache.activemq.thread.PooledTaskRunner.runTask
>> (PooledTaskRunner.java:111)
>> at
>> org.apache.activemq.thread.PooledTaskRunner.access$100
>> (PooledTaskRunner.java:26)
>> at
>> org.apache.activemq.thread.PooledTaskRunner$1.run
>> (PooledTaskRunner.java:44)
>> at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
>> $Worker.runTask(ThreadPoolExecutor.java:650)
>> at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
>> $Worker.run(ThreadPoolExecutor.java:675)
>> at java.lang.Thread.run(Thread.java:613)
>> "Thread-3":
>> at
>> org.apache.activemq.transport.MutexTransport.oneway
>> (MutexTransport.java:43)
>> - waiting to lock <0x26b8e848> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.oneway
>> (ResponseCorrelator.java:60)
>> at
>> org.apache.activemq.broker.TransportConnection.dispatch
>> (TransportConnection.java:1051)
>> at
>> org.apache.activemq.broker.TransportConnection$1.onCommand
>> (TransportConnection.java:179)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.onCommand
>> (ResponseCorrelator.java:95)
>> at
>> org.apache.activemq.transport.TransportFilter.onCommand
>> (TransportFilter.java:65)
>> at
>> org.apache.activemq.transport.vm.VMTransport.syncOneWay
>> (VMTransport.java:96)
>> at org.apache.activemq.transport.vm.VMTransport.oneway
>> (VMTransport.java:83)
>> at
>> org.apache.activemq.transport.MutexTransport.oneway
>> (MutexTransport.java:43)
>> - locked <0x26b8fa00> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.asyncRequest
>> (ResponseCorrelator.java:69)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.request
>> (ResponseCorrelator.java:74)
>> at
>> org.apache.activemq.ActiveMQConnection.syncSendPacket
>> (ActiveMQConnection.java:1171)
>> at
>> org.apache.activemq.TransactionContext.commit
>> (TransactionContext.java:260)
>> at
>> org.apache.activemq.ra.LocalAndXATransaction.commit
>> (LocalAndXATransaction.java:58)
>> at
>> org.jencks.LocalTransactionEndpoint.afterDelivery
>> (LocalTransactionEndpoint.java:52)
>> at
>> org.apache.activemq.ra.MessageEndpointProxy
>> $MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
>> at
>> org.apache.activemq.ra.MessageEndpointProxy.afterDelivery
>> (MessageEndpointProxy.java:65)
>> at
>> org.apache.activemq.ra.ServerSessionImpl.afterDelivery
>> (ServerSessionImpl.java:216)
>> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
>> at org.apache.activemq.ra.ServerSessionImpl.run
>> (ServerSessionImpl.java:165)
>> at
>> org.apache.geronimo.connector.work.WorkerContext.run
>> (WorkerContext.java:291)
>> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
>> Source)
>> at java.lang.Thread.run(Thread.java:613)
>>
>> Found 1 deadlock.
>>
>>
>> Attached there's my spring config file. I have been trying switching
>> async dispatches and sends off, thinking that could have been the
>> problem
>> but that
>> didn't help. turning off message persistence doesn't change
>> anything either.
>>
>>
>> Needless to say that the whole of this is pretty frustrating, in
>> the last
>> couple of days I have been
>> dealing with deadlocks all over the place.
>> I'm running AMC 4.1, jencks 1.3 spring 2.0.
>>
>> I hope someone will help on this, otherwise I'll probably have to
>> drop the
>> whole project:(
>>
>> thanks
>> Francesco Vivoli
>>
>> http://www.nabble.com/file/4735/container.xml container.xml
>> --
>> View this message in context: http://www.nabble.com/Random-
>> deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-
>> tf2828116.html#a7894874
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7926348
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Random deadlocks or lost messages with AMQ 4.1 and Jencks
Posted by Rob Davies <ra...@gmail.com>.
As you're using the vm://transport - you could set the async flag =
true - which will cure this problem - e.g. your connection url would
be vm://localhost?async=true
cheers,
Rob
On 15 Dec 2006, at 16:41, drvillo wrote:
>
> Hi there
>
> I have spent the whole day now tracking down this, and I have no clue.
> I have stripped everything down to a silly system:
> sender1->queue1->listener1->sender2->queue2->listener2
>
>
> I have a test that basically sends 10 messages with sender1, no more.
>
> The listeners are just logging the incoming messages, and listener1
> has
> a sender himself, to resend the same message to the second queue.
>
>
> Actually the situation is the following
> -sometimes I don't manage to send all the messages, because of a
> deadlock
> (given below)
> -sometimes I get no deadlocks but I loose a message (always the
> second) (and
> from
> the jconsole I can see that there's a message stuck in the queue
> and not
> consumed)
> -sometimes it goes fine
>
>
> the deadlock is reported below, with the last logging information
> appeared
>
> DEBUG org.apache.activemq.broker.region.AbstractRegion - Removing
> consumer:
> ID:obimac.local-51149-1166199529487-2:2:-1:4
> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - run loop end
> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - Session
> returned to
> pool: ServerSessionImpl:1
> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Run finished
> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Work completed:
> javax.resource.spi.work.WorkEvent[source=Work :ServerSessionImpl:1]
> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - ServerSession
> requested.
>
>
> ...
>
> Found one Java-level deadlock:
> =============================
> "ActiveMQ Task":
> waiting to lock monitor 0x018123fc (object 0x26b8fa00, a
> java.lang.Object),
> which is held by "Thread-3"
> "Thread-3":
> waiting to lock monitor 0x018123d8 (object 0x26b8e848, a
> java.lang.Object),
> which is held by "ActiveMQ Task"
>
> Java stack information for the threads listed above:
> ===================================================
> "ActiveMQ Task":
> at
> org.apache.activemq.transport.MutexTransport.oneway
> (MutexTransport.java:43)
> - waiting to lock <0x26b8fa00> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway
> (ResponseCorrelator.java:60)
> at
> org.apache.activemq.ActiveMQConnection.asyncSendPacket
> (ActiveMQConnection.java:1151)
> at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:
> 226)
> at
> org.apache.activemq.ActiveMQConnection.createSession
> (ActiveMQConnection.java:277)
> at
> org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl
> (ServerSessionPoolImpl.java:63)
> at
> org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession
> (ServerSessionPoolImpl.java:116)
> at
> org.apache.activemq.ActiveMQConnectionConsumer.dispatch
> (ActiveMQConnectionConsumer.java:136)
> at
> org.apache.activemq.ActiveMQConnection.onCommand
> (ActiveMQConnection.java:1473)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand
> (ResponseCorrelator.java:95)
> at
> org.apache.activemq.transport.TransportFilter.onCommand
> (TransportFilter.java:65)
> at
> org.apache.activemq.transport.vm.VMTransport.syncOneWay
> (VMTransport.java:96)
> at org.apache.activemq.transport.vm.VMTransport.oneway
> (VMTransport.java:83)
> at
> org.apache.activemq.transport.MutexTransport.oneway
> (MutexTransport.java:43)
> - locked <0x26b8e848> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway
> (ResponseCorrelator.java:60)
> at
> org.apache.activemq.broker.TransportConnection.dispatch
> (TransportConnection.java:1051)
> at
> org.apache.activemq.broker.TransportConnection.processDispatch
> (TransportConnection.java:772)
> at
> org.apache.activemq.broker.TransportConnection.iterate
> (TransportConnection.java:788)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask
> (PooledTaskRunner.java:111)
> at
> org.apache.activemq.thread.PooledTaskRunner.access$100
> (PooledTaskRunner.java:26)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run
> (PooledTaskRunner.java:44)
> at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:650)
> at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:613)
> "Thread-3":
> at
> org.apache.activemq.transport.MutexTransport.oneway
> (MutexTransport.java:43)
> - waiting to lock <0x26b8e848> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway
> (ResponseCorrelator.java:60)
> at
> org.apache.activemq.broker.TransportConnection.dispatch
> (TransportConnection.java:1051)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand
> (TransportConnection.java:179)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand
> (ResponseCorrelator.java:95)
> at
> org.apache.activemq.transport.TransportFilter.onCommand
> (TransportFilter.java:65)
> at
> org.apache.activemq.transport.vm.VMTransport.syncOneWay
> (VMTransport.java:96)
> at org.apache.activemq.transport.vm.VMTransport.oneway
> (VMTransport.java:83)
> at
> org.apache.activemq.transport.MutexTransport.oneway
> (MutexTransport.java:43)
> - locked <0x26b8fa00> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.asyncRequest
> (ResponseCorrelator.java:69)
> at
> org.apache.activemq.transport.ResponseCorrelator.request
> (ResponseCorrelator.java:74)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket
> (ActiveMQConnection.java:1171)
> at
> org.apache.activemq.TransactionContext.commit
> (TransactionContext.java:260)
> at
> org.apache.activemq.ra.LocalAndXATransaction.commit
> (LocalAndXATransaction.java:58)
> at
> org.jencks.LocalTransactionEndpoint.afterDelivery
> (LocalTransactionEndpoint.java:52)
> at
> org.apache.activemq.ra.MessageEndpointProxy
> $MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
> at
> org.apache.activemq.ra.MessageEndpointProxy.afterDelivery
> (MessageEndpointProxy.java:65)
> at
> org.apache.activemq.ra.ServerSessionImpl.afterDelivery
> (ServerSessionImpl.java:216)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
> at org.apache.activemq.ra.ServerSessionImpl.run
> (ServerSessionImpl.java:165)
> at
> org.apache.geronimo.connector.work.WorkerContext.run
> (WorkerContext.java:291)
> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:613)
>
> Found 1 deadlock.
>
>
> Attached there's my spring config file. I have been trying switching
> async dispatches and sends off, thinking that could have been the
> problem
> but that
> didn't help. turning off message persistence doesn't change
> anything either.
>
>
> Needless to say that the whole of this is pretty frustrating, in
> the last
> couple of days I have been
> dealing with deadlocks all over the place.
> I'm running AMC 4.1, jencks 1.3 spring 2.0.
>
> I hope someone will help on this, otherwise I'll probably have to
> drop the
> whole project:(
>
> thanks
> Francesco Vivoli
>
> http://www.nabble.com/file/4735/container.xml container.xml
> --
> View this message in context: http://www.nabble.com/Random-
> deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-
> tf2828116.html#a7894874
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Random deadlocks or lost messages with AMQ 4.1 and Jencks
Posted by drvillo <f....@gmail.com>.
Hi
thanks for the hint, getting the HEAD revision solves that:)
gnodet wrote:
>
> You should try with svn head.
> Your second problem may be related to
> http://issues.apache.org/activemq/browse/AMQ-1078
> which has been fixed in 4.2 and 4.1 branch (future 4.1.1).
>
> On 12/15/06, drvillo <f....@gmail.com> wrote:
>>
>> Hi there
>>
>> I have spent the whole day now tracking down this, and I have no clue.
>> I have stripped everything down to a silly system:
>> sender1->queue1->listener1->sender2->queue2->listener2
>>
>>
>> I have a test that basically sends 10 messages with sender1, no more.
>>
>> The listeners are just logging the incoming messages, and listener1 has
>> a sender himself, to resend the same message to the second queue.
>>
>>
>> Actually the situation is the following
>> -sometimes I don't manage to send all the messages, because of a deadlock
>> (given below)
>> -sometimes I get no deadlocks but I loose a message (always the second)
>> (and
>> from
>> the jconsole I can see that there's a message stuck in the queue and not
>> consumed)
>> -sometimes it goes fine
>>
>>
>> the deadlock is reported below, with the last logging information
>> appeared
>>
>> DEBUG org.apache.activemq.broker.region.AbstractRegion - Removing
>> consumer:
>> ID:obimac.local-51149-1166199529487-2:2:-1:4
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - run loop end
>> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - Session returned to
>> pool: ServerSessionImpl:1
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Run finished
>> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Work completed:
>> javax.resource.spi.work.WorkEvent[source=Work :ServerSessionImpl:1]
>> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - ServerSession
>> requested.
>>
>>
>> ...
>>
>> Found one Java-level deadlock:
>> =============================
>> "ActiveMQ Task":
>> waiting to lock monitor 0x018123fc (object 0x26b8fa00, a
>> java.lang.Object),
>> which is held by "Thread-3"
>> "Thread-3":
>> waiting to lock monitor 0x018123d8 (object 0x26b8e848, a
>> java.lang.Object),
>> which is held by "ActiveMQ Task"
>>
>> Java stack information for the threads listed above:
>> ===================================================
>> "ActiveMQ Task":
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
>> - waiting to lock <0x26b8fa00> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> at
>> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1151)
>> at
>> org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:226)
>> at
>> org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:277)
>> at
>> org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl(ServerSessionPoolImpl.java:63)
>> at
>> org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:116)
>> at
>> org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:136)
>> at
>> org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1473)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
>> at
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
>> at
>> org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
>> at
>> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
>> - locked <0x26b8e848> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> at
>> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
>> at
>> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:772)
>> at
>> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:788)
>> at
>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:111)
>> at
>> org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
>> at
>> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44)
>> at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>> at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>> at java.lang.Thread.run(Thread.java:613)
>> "Thread-3":
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
>> - waiting to lock <0x26b8e848> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
>> at
>> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
>> at
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
>> at
>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
>> at
>> org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
>> at
>> org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
>> - locked <0x26b8fa00> (a java.lang.Object)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
>> at
>> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74)
>> at
>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1171)
>> at
>> org.apache.activemq.TransactionContext.commit(TransactionContext.java:260)
>> at
>> org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:58)
>> at
>> org.jencks.LocalTransactionEndpoint.afterDelivery(LocalTransactionEndpoint.java:52)
>> at
>> org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
>> at
>> org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
>> at
>> org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:216)
>> at
>> org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
>> at
>> org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
>> at
>> org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
>> at
>> EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
>> Source)
>> at java.lang.Thread.run(Thread.java:613)
>>
>> Found 1 deadlock.
>>
>>
>> Attached there's my spring config file. I have been trying switching
>> async dispatches and sends off, thinking that could have been the problem
>> but that
>> didn't help. turning off message persistence doesn't change anything
>> either.
>>
>>
>> Needless to say that the whole of this is pretty frustrating, in the last
>> couple of days I have been
>> dealing with deadlocks all over the place.
>> I'm running AMC 4.1, jencks 1.3 spring 2.0.
>>
>> I hope someone will help on this, otherwise I'll probably have to drop
>> the
>> whole project:(
>>
>> thanks
>> Francesco Vivoli
>>
>> http://www.nabble.com/file/4735/container.xml container.xml
>> --
>> View this message in context:
>> http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7894874
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Cheers,
> Guillaume Nodet
>
>
--
View this message in context: http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7945235
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Random deadlocks or lost messages with AMQ 4.1 and Jencks
Posted by Guillaume Nodet <gn...@gmail.com>.
You should try with svn head.
Your second problem may be related to
http://issues.apache.org/activemq/browse/AMQ-1078
which has been fixed in 4.2 and 4.1 branch (future 4.1.1).
On 12/15/06, drvillo <f....@gmail.com> wrote:
>
> Hi there
>
> I have spent the whole day now tracking down this, and I have no clue.
> I have stripped everything down to a silly system:
> sender1->queue1->listener1->sender2->queue2->listener2
>
>
> I have a test that basically sends 10 messages with sender1, no more.
>
> The listeners are just logging the incoming messages, and listener1 has
> a sender himself, to resend the same message to the second queue.
>
>
> Actually the situation is the following
> -sometimes I don't manage to send all the messages, because of a deadlock
> (given below)
> -sometimes I get no deadlocks but I loose a message (always the second) (and
> from
> the jconsole I can see that there's a message stuck in the queue and not
> consumed)
> -sometimes it goes fine
>
>
> the deadlock is reported below, with the last logging information appeared
>
> DEBUG org.apache.activemq.broker.region.AbstractRegion - Removing consumer:
> ID:obimac.local-51149-1166199529487-2:2:-1:4
> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - run loop end
> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - Session returned to
> pool: ServerSessionImpl:1
> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Run finished
> DEBUG org.apache.activemq.ra.ServerSessionImpl:1 - Work completed:
> javax.resource.spi.work.WorkEvent[source=Work :ServerSessionImpl:1]
> DEBUG org.apache.activemq.ra.ServerSessionPoolImpl - ServerSession
> requested.
>
>
> ...
>
> Found one Java-level deadlock:
> =============================
> "ActiveMQ Task":
> waiting to lock monitor 0x018123fc (object 0x26b8fa00, a
> java.lang.Object),
> which is held by "Thread-3"
> "Thread-3":
> waiting to lock monitor 0x018123d8 (object 0x26b8e848, a
> java.lang.Object),
> which is held by "ActiveMQ Task"
>
> Java stack information for the threads listed above:
> ===================================================
> "ActiveMQ Task":
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
> - waiting to lock <0x26b8fa00> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1151)
> at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:226)
> at
> org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:277)
> at
> org.apache.activemq.ra.ServerSessionPoolImpl.createServerSessionImpl(ServerSessionPoolImpl.java:63)
> at
> org.apache.activemq.ra.ServerSessionPoolImpl.getServerSession(ServerSessionPoolImpl.java:116)
> at
> org.apache.activemq.ActiveMQConnectionConsumer.dispatch(ActiveMQConnectionConsumer.java:136)
> at
> org.apache.activemq.ActiveMQConnection.onCommand(ActiveMQConnection.java:1473)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
> at
> org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
> at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
> - locked <0x26b8e848> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
> at
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:772)
> at
> org.apache.activemq.broker.TransportConnection.iterate(TransportConnection.java:788)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:111)
> at
> org.apache.activemq.thread.PooledTaskRunner.access$100(PooledTaskRunner.java:26)
> at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:44)
> at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:613)
> "Thread-3":
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
> - waiting to lock <0x26b8e848> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
> at
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1051)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:95)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:65)
> at
> org.apache.activemq.transport.vm.VMTransport.syncOneWay(VMTransport.java:96)
> at org.apache.activemq.transport.vm.VMTransport.oneway(VMTransport.java:83)
> at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:43)
> - locked <0x26b8fa00> (a java.lang.Object)
> at
> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:69)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:74)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1171)
> at
> org.apache.activemq.TransactionContext.commit(TransactionContext.java:260)
> at
> org.apache.activemq.ra.LocalAndXATransaction.commit(LocalAndXATransaction.java:58)
> at
> org.jencks.LocalTransactionEndpoint.afterDelivery(LocalTransactionEndpoint.java:52)
> at
> org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
> at
> org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
> at
> org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:216)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:749)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at
> org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:291)
> at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Thread.java:613)
>
> Found 1 deadlock.
>
>
> Attached there's my spring config file. I have been trying switching
> async dispatches and sends off, thinking that could have been the problem
> but that
> didn't help. turning off message persistence doesn't change anything either.
>
>
> Needless to say that the whole of this is pretty frustrating, in the last
> couple of days I have been
> dealing with deadlocks all over the place.
> I'm running AMC 4.1, jencks 1.3 spring 2.0.
>
> I hope someone will help on this, otherwise I'll probably have to drop the
> whole project:(
>
> thanks
> Francesco Vivoli
>
> http://www.nabble.com/file/4735/container.xml container.xml
> --
> View this message in context: http://www.nabble.com/Random-deadlocks-or-lost-messages-with-AMQ-4.1-and-Jencks-tf2828116.html#a7894874
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
--
Cheers,
Guillaume Nodet