You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by badavis <br...@gmail.com> on 2014/07/04 15:47:23 UTC
Re: ActiveMQ message dequeuing hangs
I ran into the same issue. The good thing is I figured out what was causing
the issue. I believe the issue is two issues. There is a deadlock in the
processing of async jobs that is blocking a job that can unblock the
deadlock.
The two threads that are important are:
Thread 43139: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information
may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=20,
line=186 (Compiled frame [deoptimized])
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=7, line=834 (Compiled frame [deoptimized])
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int)
@bci=89, line=994 (Compiled frame [deoptimized])
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int)
@bci=47, line=1303 (Interpreted frame)
- java.util.concurrent.CountDownLatch.await() @bci=11, line=236
(Interpreted frame)
-
org.apache.activemq.leveldb.LevelDBStore.rollback(org.apache.activemq.command.TransactionId)
@bci=152, line=498 (Interpreted frame)
*-> org.apache.activemq.transaction.LocalTransaction.rollback() @bci=111,
line=94 (Interpreted frame)*
-
org.apache.activemq.broker.TransactionBroker.rollbackTransaction(org.apache.activemq.broker.ConnectionContext,
org.apache.activemq.command.TransactionId) @bci=15, line=258 (Interpreted
frame)
-
org.apache.activemq.broker.BrokerFilter.rollbackTransaction(org.apache.activemq.broker.ConnectionContext,
org.apache.activemq.command.TransactionId) @bci=12, line=142 (Interpreted
frame)
-
org.apache.activemq.broker.MutableBrokerFilter.rollbackTransaction(org.apache.activemq.broker.ConnectionContext,
org.apache.activemq.command.TransactionId) @bci=12, line=147 (Interpreted
frame)
-
org.apache.activemq.broker.TransportConnection.processRollbackTransaction(org.apache.activemq.command.TransactionInfo)
@bci=44, line=442 (Interpreted frame)
-
org.apache.activemq.command.TransactionInfo.visit(org.apache.activemq.state.CommandVisitor)
@bci=146, line=104 (Compiled frame [deoptimized])
-
org.apache.activemq.broker.TransportConnection.service(org.apache.activemq.command.Command)
@bci=52, line=292 (Compiled frame [deoptimized])
-
org.apache.activemq.broker.TransportConnection$1.onCommand(java.lang.Object)
@bci=95, line=149 (Compiled frame [deoptimized])
- org.apache.activemq.transport.MutexTransport.onCommand(java.lang.Object)
@bci=58, line=50 (Compiled frame [deoptimized])
-
org.apache.activemq.transport.WireFormatNegotiator.onCommand(java.lang.Object)
@bci=35, line=113 (Compiled frame [deoptimized])
-
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(java.lang.Object)
@bci=162, line=270 (Compiled frame [deoptimized])
-
org.apache.activemq.transport.TransportSupport.doConsume(java.lang.Object)
@bci=27, line=83 (Compiled frame [deoptimized])
- org.apache.activemq.transport.tcp.TcpTransport.doRun() @bci=13, line=214
(Compiled frame [deoptimized])
- org.apache.activemq.transport.tcp.TcpTransport.run() @bci=70, line=196
(Compiled frame [deoptimized])
- java.lang.Thread.run() @bci=17, line=724 (Interpreted frame)
Thread 38184: (state = BLOCKED)
* -> org.apache.activemq.leveldb.LevelDBStore.getTopicGCPositions() @bci=15,
line=768 (Interpreted frame)*
- org.apache.activemq.leveldb.DBManager$$anonfun$pollGc$1.apply$mcV$sp()
@bci=28, line=640 (Interpreted frame)
- org.fusesource.hawtdispatch.package$$anon$4.run() @bci=10, line=357
(Compiled frame [deoptimized])
- org.fusesource.hawtdispatch.internal.SerialDispatchQueue.run() @bci=317,
line=100 (Compiled frame [deoptimized])
- org.fusesource.hawtdispatch.internal.pool.SimpleThread.run() @bci=105,
line=77 (Compiled frame [deoptimized])
I bolded and -> the lines that are important with the deadlock. The
rollback thread holds the lock of the message store and the
getTopicGCPositions needs to get the lock of the message store (the
identified lines in the stack trace). That may not seem that important but
the more important thing is where the getTopicGCPositions is running. It is
running in the hawtdispatcher. If you could look at the SerialDispatchQueue
you would see that it tries to run all the jobs in one run. Well this is a
problem if one of the front jobs is blocked until a job behind it runs.
That is where we are. The job that the rollback is waiting on is scheduled
to run behind the pollGc job.
My belief is their are two issues:
* the LevelDBStore class does not fully own its lock
* the hawtdispatcher does not distribute the jobs over the dispatcher's
threads, but chooses to run many jobs in one thread when other threads are
available.
Hope this helps.
You have a good one.
Brian
P.S. The only mitigation is a code change. You cannot change the polling
interval to lower the likely hood of it happening because it is hard coded
to every 10 seconds.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4682806.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ message dequeuing hangs
Posted by mattmadhavan <ma...@yahoo.com>.
Hi James,
Yes I did yesterday and facing the same issue as I did with KahaDB. Problem
is no errors, warnings nothing!
Not sure whats happening!
Any insight into this will be appreciated.
Thanks
Matt'M
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4694433.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ message dequeuing hangs
Posted by jameshome <ja...@googlemail.com>.
Hi Matt,
The previous issue (AMQ-5125) was specifically an issue when using LevelDB,
so if you're using KahaDB it won't be exactly the same issue, despite the
similarity of the symptoms.
Out of interest, have you tried using LevelDB? Do you get the same issue?
James
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4694406.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ message dequeuing hangs
Posted by mattmadhavan <ma...@yahoo.com>.
HI Skate,
Thanks for the reply. I actually started with 5.10 when I ran into this
problem. I switched to the latest ActiveMQ 5.11.1 still the same issue.
It actually hung once more just I while I am writing this reply. Very
frustrating.
Any ideas please?
I appreciate your help in advance!
Thanks
Matt'M
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4694393.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ message dequeuing hangs
Posted by skate056 <sk...@gmail.com>.
Have you tried this on 5.10? I believe James had committed a fix for that.
Thanks
Saket
On Mon, 6 Apr 2015 at 22:23, mattmadhavan [via ActiveMQ] <
ml-node+s2283324n4694366h99@n4.nabble.com> wrote:
> HI James,
> I am trying to get my ActiveMQ configuration closer to yours. Can you
> please elaborate what you mean by:
>
> "*I modified LevelDBStore to use a private lock instead of this*" and
> how do I do it? I been stuck on this issue for couple of weeks now.
>
> Thanks
> Matt'M
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4694366.html
> To unsubscribe from ActiveMQ message dequeuing hangs, click here
> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4681366&code=c2thdGUwNTZAZ21haWwuY29tfDQ2ODEzNjZ8MzM0NzIxODA=>
> .
> NAML
> <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4694367.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ message dequeuing hangs
Posted by mattmadhavan <ma...@yahoo.com>.
HI James,
I am trying to get my ActiveMQ configuration closer to yours. Can you please
elaborate what you mean by:
"*I modified LevelDBStore to use a private lock instead of this*" and how
do I do it? I been stuck on this issue for couple of weeks now.
Thanks
Matt'M
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4694366.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ message dequeuing hangs
Posted by mattmadhavan <ma...@yahoo.com>.
Hello,
I am facing almost identical problem. I am using the latest AMQ 5.11.1 with
KhaDB.
Any suggestions please?
Thanks
Matt
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4694306.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ message dequeuing hangs
Posted by jameshome <ja...@googlemail.com>.
Based on your post Brian, I modified LevelDBStore to use a private lock
instead of _this_, and it does indeed solve the issue.
I've attached a 5.9.2 patch to:
https://issues.apache.org/jira/browse/AMQ-5125
Thanks for your help,
James
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-message-dequeuing-hangs-tp4681366p4682880.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.