You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by sbarlabanov <co...@sergiy-barlabanov.de> on 2015/03/24 22:24:27 UTC
MDB stops consuming messages
Hi,
this is the continuation of my forum entry
http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-0-queue-slowed-down-restart-helped-td4690706.html#a4690723.
But since the discussion there went to another direction I start a new
topic.
We got the issue again several times in the last two month.
More details: we have a standalone running ActiveMQ 5.10.0, Glassfish
3.1.2.2 with ActiveMQ RAR and an MDB consuming messages from an ActiveMQ
queue.
Suddenly after weeks successful processing the MDB stops processing any
message (I mentioned that already). In jconsole we see the corresponding JMS
consumer on the queue. MessageCountAwaitingAcknowledge of the consumer is
equal the prefetch buffer (10 in our case). From time to time a new consumer
occurs on the queue (this is seen in jconsole), but the first one stays with
MessageCountAwaitingAcknowledge=prefetch buffer.
When we stop Glassfish, the following interesting exception appears in
activemq.log:
java.lang.IllegalStateException: Cannot remove session from connection that
had not been registered: ID:NOEUD-SI2-55792-1427222776632-47:1
at
org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:677)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
at
org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:74)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
at
org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
at
org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
at
org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
at
org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_45]
at java.lang.Thread.run(Thread.java:744)[:1.7.0_45]
It seems that something goes wrong in ActiveMQ RAR. Seems that something
hangs around in the RAR locking the messages. Because otherwise ActiveMQ
will not complain about the session which is going to be removed from an
unregistered session. So something went wrong before. And the broker did not
get it and still thinks that the consumer/session/connection is open.
--
View this message in context: http://activemq.2283324.n4.nabble.com/MDB-stops-consuming-messages-tp4693769.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: MDB stops consuming messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
Oh, I forgot: was there any indication of Producer Flow Control kicking in
on the broker prior to when the consumer stopped processing messages?
On Sun, Apr 26, 2015 at 9:06 AM, Tim Bain <tb...@alumni.duke.edu> wrote:
> Did you ever solve this issue?
>
> If not, can you please clarify a couple things in your description?
>
> 1. Did you confirm (via a thread dump) that the MDB wasn't actually
> hung somewhere in your consumer code? Clearly the broker believed it had
> dispatched a prefetch buffer worth of messages to the consumer, so the
> question is whether the consumer's ActiveMQ client/RAR code had attempted
> to dispatch the next message to your MDB or not. And if the thread wasn't
> stuck in your MDB code, where was it stuck instead?
> 2. When you shut down Glassfish, was your MDB consumer connected or
> disconnected? The first half of your description sounded like it was
> connected, while the last paragraph sounded like it was disconnected.
> 3. When you connected the additional consumer, did it process messages
> as expected? You never said what happened when those additional consumers
> connected.
> 4. Did that ID match one of the consumers (or one of their
> connections; I can never tell consumer IDs and connection IDs apart)? If
> so, which one?
> 5. If you can reproduce this reliably, does the exception happen if
> you shut down Glassfish without attempting to connect any additional
> consumers?
>
> Tim
>
> On Tue, Mar 24, 2015 at 3:24 PM, sbarlabanov <contact@sergiy-barlabanov.de
> > wrote:
>
>> Hi,
>>
>> this is the continuation of my forum entry
>>
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-0-queue-slowed-down-restart-helped-td4690706.html#a4690723
>> .
>> But since the discussion there went to another direction I start a new
>> topic.
>> We got the issue again several times in the last two month.
>> More details: we have a standalone running ActiveMQ 5.10.0, Glassfish
>> 3.1.2.2 with ActiveMQ RAR and an MDB consuming messages from an ActiveMQ
>> queue.
>> Suddenly after weeks successful processing the MDB stops processing any
>> message (I mentioned that already). In jconsole we see the corresponding
>> JMS
>> consumer on the queue. MessageCountAwaitingAcknowledge of the consumer is
>> equal the prefetch buffer (10 in our case). From time to time a new
>> consumer
>> occurs on the queue (this is seen in jconsole), but the first one stays
>> with
>> MessageCountAwaitingAcknowledge=prefetch buffer.
>> When we stop Glassfish, the following interesting exception appears in
>> activemq.log:
>>
>> java.lang.IllegalStateException: Cannot remove session from connection
>> that
>> had not been registered: ID:NOEUD-SI2-55792-1427222776632-47:1
>> at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:677)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
>> at
>>
>> org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:74)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>> at
>>
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
>> at
>>
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
>> at
>>
>> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>> at
>>
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>> at
>>
>> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>> at
>>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>> at
>>
>> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>> at
>>
>> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>> at
>>
>> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>> at
>>
>> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_45]
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_45]
>> at java.lang.Thread.run(Thread.java:744)[:1.7.0_45]
>>
>> It seems that something goes wrong in ActiveMQ RAR. Seems that something
>> hangs around in the RAR locking the messages. Because otherwise ActiveMQ
>> will not complain about the session which is going to be removed from an
>> unregistered session. So something went wrong before. And the broker did
>> not
>> get it and still thinks that the consumer/session/connection is open.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/MDB-stops-consuming-messages-tp4693769.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
Re: MDB stops consuming messages
Posted by Tim Bain <tb...@alumni.duke.edu>.
Did you ever solve this issue?
If not, can you please clarify a couple things in your description?
1. Did you confirm (via a thread dump) that the MDB wasn't actually hung
somewhere in your consumer code? Clearly the broker believed it had
dispatched a prefetch buffer worth of messages to the consumer, so the
question is whether the consumer's ActiveMQ client/RAR code had attempted
to dispatch the next message to your MDB or not. And if the thread wasn't
stuck in your MDB code, where was it stuck instead?
2. When you shut down Glassfish, was your MDB consumer connected or
disconnected? The first half of your description sounded like it was
connected, while the last paragraph sounded like it was disconnected.
3. When you connected the additional consumer, did it process messages
as expected? You never said what happened when those additional consumers
connected.
4. Did that ID match one of the consumers (or one of their connections;
I can never tell consumer IDs and connection IDs apart)? If so, which one?
5. If you can reproduce this reliably, does the exception happen if you
shut down Glassfish without attempting to connect any additional consumers?
Tim
On Tue, Mar 24, 2015 at 3:24 PM, sbarlabanov <co...@sergiy-barlabanov.de>
wrote:
> Hi,
>
> this is the continuation of my forum entry
>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-0-queue-slowed-down-restart-helped-td4690706.html#a4690723
> .
> But since the discussion there went to another direction I start a new
> topic.
> We got the issue again several times in the last two month.
> More details: we have a standalone running ActiveMQ 5.10.0, Glassfish
> 3.1.2.2 with ActiveMQ RAR and an MDB consuming messages from an ActiveMQ
> queue.
> Suddenly after weeks successful processing the MDB stops processing any
> message (I mentioned that already). In jconsole we see the corresponding
> JMS
> consumer on the queue. MessageCountAwaitingAcknowledge of the consumer is
> equal the prefetch buffer (10 in our case). From time to time a new
> consumer
> occurs on the queue (this is seen in jconsole), but the first one stays
> with
> MessageCountAwaitingAcknowledge=prefetch buffer.
> When we stop Glassfish, the following interesting exception appears in
> activemq.log:
>
> java.lang.IllegalStateException: Cannot remove session from connection that
> had not been registered: ID:NOEUD-SI2-55792-1427222776632-47:1
> at
>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:677)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
> at
>
> org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:74)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
> at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
> at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
> at
>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
> at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
> at
>
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
> at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
> at
>
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
> at
>
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
> at
>
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
> at
>
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_45]
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744)[:1.7.0_45]
>
> It seems that something goes wrong in ActiveMQ RAR. Seems that something
> hangs around in the RAR locking the messages. Because otherwise ActiveMQ
> will not complain about the session which is going to be removed from an
> unregistered session. So something went wrong before. And the broker did
> not
> get it and still thinks that the consumer/session/connection is open.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/MDB-stops-consuming-messages-tp4693769.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>