You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Luke Noel-Storr <lu...@integrate.co.uk> on 2012/05/08 18:27:18 UTC
Messages stuck in pending
I have a problem where messages seem to be getting stuck in a pending state.
Initially it seemed they only got stuck in this state if there was no
consumer when the message was sent, and even after a consumer was started,
they just sat there. They then seemingly caused all subsequent messages to
get stuck in a pending state too. Purging the queue seemed to allow
messages to get through. My colleagues also had this same problem (on
separate systems).
This was frustrating enough, but now it seems every message being sent goes
to pending, and gets no further. Purging the queue nor restarting activemq
seem to help. I seemed to manage to send one message successfully after a
full machine reboot.
We're sending and receiving messages via JMS.
I'm using 5.5.0. My colleagues have 5.5.1 (I think).
I'm not sure what other information may help diagnose this issue, but let me
know, and I'll do my best to supply it.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-tp4617979.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in pending
Posted by Luke Noel-Storr <lu...@integrate.co.uk>.
I finally managed to get to the bottom of this one.
The problem seems to have been that the sending machine's clock (which is
also where the broker is) was 13 minutes behind the receiving machine's
clock and messages were being sent with a TTL of 10 minutes. This seemingly
caused them to just sit in pending, until they then expired.
Increasing the TTL to 20 minutes solved the issue, as did manually adjusting
the time on the servers so that they were the same.
Both servers were set up to synchronise their clocks to an ntp server, but
it seems the relevant ports may be blocked on our network.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-tp4617979p4653771.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in pending
Posted by Luke Noel-Storr <lu...@integrate.co.uk>.
I've also tried removing the filter/selector from the receiver, and it still
seems to receive nothing.
I'm sure it's just something simple and dumb that has suddenly caused this
problem, but I can't figure out what.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-tp4617979p4632003.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in pending
Posted by Luke Noel-Storr <lu...@integrate.co.uk>.
I've added logging in the receiver and it never seems to receive the message.
Messages are all auto-acknowledge.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-tp4617979p4631844.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in pending
Posted by Gary Tully <ga...@gmail.com>.
from jmx. it looks like that consumer has got the message and not
acked it, from the broker perspective it has one message that has a
pending ack, so that message is only available to the consumer that
received it until that consumer closes or acks.
Can you close that consumers connection via jmx and see if the message
gets dispatched to another?
What is the ack mode for the consumers?
On 11 May 2012 10:44, Luke Noel-Storr <lu...@integrate.co.uk> wrote:
> There are 5 consumers on the queue.
>
> They each have a different filter, but I have checked the filters against
> stuck messages, and there are consumer that should match each message.
>
> There is not heavy load, it gets stuck with just one message sent.
>
> If I view consumers in the web admin, the one which should be matching the
> stuck messages shows:
>
> Enqueues: 1
> Dequeues: 0
> Dispatched: 1
> Dispatched Queue: 1
>
> The problem seemed to go away yesterday, but is now back again.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-tp4617979p4625973.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
http://fusesource.com
http://blog.garytully.com
Re: Messages stuck in pending
Posted by Luke Noel-Storr <lu...@integrate.co.uk>.
the selector is:
destinationHostId = '17d7400f-3623-4279-a9d5-5bbb2a56e573' and MESSAGE_TYPE
is null
Message properties are:
destinationHostId 17d7400f-3623-4279-a9d5-5bbb2a56e573
It's nothing complex.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-tp4617979p4626162.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in pending
Posted by mickhayes <mi...@gmail.com>.
Can you show us your filters and an example message that you expect to match
a filter?
-----
Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-tp4617979p4626026.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in pending
Posted by Luke Noel-Storr <lu...@integrate.co.uk>.
There are 5 consumers on the queue.
They each have a different filter, but I have checked the filters against
stuck messages, and there are consumer that should match each message.
There is not heavy load, it gets stuck with just one message sent.
If I view consumers in the web admin, the one which should be matching the
stuck messages shows:
Enqueues: 1
Dequeues: 0
Dispatched: 1
Dispatched Queue: 1
The problem seemed to go away yesterday, but is now back again.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-tp4617979p4625973.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages stuck in pending
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Luke,
could you check the number of consumer on the queue ?
I saw a similar issue where my consumer was present while the load was
not too high, and on heavy load, the consumer disappears and so the
messages stayed in pending.
Regards
JB
On 05/08/2012 06:27 PM, Luke Noel-Storr wrote:
> I have a problem where messages seem to be getting stuck in a pending state.
>
> Initially it seemed they only got stuck in this state if there was no
> consumer when the message was sent, and even after a consumer was started,
> they just sat there. They then seemingly caused all subsequent messages to
> get stuck in a pending state too. Purging the queue seemed to allow
> messages to get through. My colleagues also had this same problem (on
> separate systems).
>
> This was frustrating enough, but now it seems every message being sent goes
> to pending, and gets no further. Purging the queue nor restarting activemq
> seem to help. I seemed to manage to send one message successfully after a
> full machine reboot.
>
> We're sending and receiving messages via JMS.
>
> I'm using 5.5.0. My colleagues have 5.5.1 (I think).
>
>
> I'm not sure what other information may help diagnose this issue, but let me
> know, and I'll do my best to supply it.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-tp4617979.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com