You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@activemq.apache.org by ByteFlinger <by...@gmail.com> on 2012/07/19 15:56:17 UTC

ActiveMQ not expiring all messages and leaving them stuck in queue?

I am having a small issue with activemq.

I have a queue which I send messages to with a certain TTL. When these
messages expire, once the queue is read I expect all expired messages to be
moved towards ActiveMQ.DLQ however what is happening is that most will do
that however a few (Usually between 1 to 5) just sit in the queue and will
not move.

You can see that the queue count on the web console is showing some messages
in that queue and you can see the same exact different between enqueued and
dequeued messages. Strangely when clicking in the queue in the web console
shows no messages whatsoever (An empty view) even though the count shows
they are there (and I know they are there because they never appear in my
logs which listen and logs all messages in ActiveMQ.DLQ).

This is happening consistently. Does anybody know what is causing this? It
feels to me that it is a bug in activemq however I cannot find any
information on it when searching around the net and debug logs don't tell me
much other than the line below:

(It can be seen above that enqueued and dequeued difference is 4
and that is the number that is appearing in the pending column for that
queue however clicking on the queue shows no messages).

Any help or insight is appreciated. Should I file a bug?

--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-not-expiring-all-messages-and-leaving-them-stuck-in-queue-tp4654172.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ not expiring all messages and leaving them stuck in queue?

Posted by ByteFlinger <by...@gmail.com>.

I have written a simple unit test today however noticed the issue was not
happening, that's when I brought the number of consumers in our code to 1
and then it worked there too. It seems that it only happens if you have more
than 1 consumer.

I will see if I can thread the unit test to create more consumers (Though I
am not entirely certain how to do that) and recreate the issue.



--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-not-expiring-all-messages-and-leaving-them-stuck-in-queue-tp4654172p4654195.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ not expiring all messages and leaving them stuck in queue?

Posted by ByteFlinger <by...@gmail.com>.

As I corrected on the discussion thread I have tested on both 5.6.0 and
5.4.3.

I am in fact using a test case to test this however due to the complexity
of the project it will not be of much use here. If I find a little time to
write a simple test at some point I'll be sure to post it here.

On the example above the queue contains 20 messages, 10 of which contains a
short ttl (10 seconds) and 10 with a longer ttl (10 minutes). After waiting
a little over 10 seconds I start some consumers in the queue which will
deliver 10 messages to its destination and (supposedly) 10 messages to the
DLQ however out of the 10 expired messages only about 7 move out, the rest
just sit there stuck. Every time I run the test the amount of "stuck"
messages may increase as more and more expired messages are not delivered.

I am not sure what you mean by unacked but if it helps, I believe the
consumers are actually always running but to be able to test I place a
temporary block on the destination ip so that the messages are not consumed
until they have had time to expire. Surely if some expire, all should?
On Jul 19, 2012 5:32 PM, "gtully [via ActiveMQ]" <
ml-node+s2283324n4654179h67@n4.nabble.com> wrote:

> Do you think you could create a test case that can reproduce this
> behavior? If so, please open a jira issue.
> I wonder if this is related to inflight=20, those messages are
> dispatched and unacked by some consumer, so they may not expire in
> that scenario.
> Are you on 5.6, if not it would be good to see if you can reproduce
> with 5.6 or even better 5.7-SNAPSHOT (which is trunk).
>
> On 19 July 2012 14:56, ByteFlinger <[hidden email]<http://user/SendEmail.jtp?type=node&node=4654179&i=0>>
> wrote:
>
> > I am having a small issue with activemq.
> >
> > I have a queue which I send messages to with a certain TTL. When these
> > messages expire, once the queue is read I expect all expired messages to
> be
> > moved towards ActiveMQ.DLQ however what is happening is that most will
> do
> > that however a few (Usually between 1 to 5) just sit in the queue and
> will
> > not move.
> >
> > You can see that the queue count on the web console is showing some
> messages
> > in that queue and you can see the same exact different between enqueued
> and
> > dequeued messages. Strangely when clicking in the queue in the web
> console
> > shows no messages whatsoever (An empty view) even though the count shows
> > they are there (and I know they are there because they never appear in
> my
> > logs which listen and logs all messages in ActiveMQ.DLQ).
> >
> > This is happening consistently. Does anybody know what is causing this?
> It
> > feels to me that it is a bug in activemq however I cannot find any
> > information on it when searching around the net and debug logs don't
> tell me
> > much other than the line below:
> >
> > 2012-07-19 15:42:43,417 | DEBUG | Expiring messages .. |
> > org.apache.activemq.broker.region.Queue | ActiveMQ Broker[localhost]
> > Scheduler
> > 2012-07-19 15:42:43,417 | DEBUG | SomeQueue toPageIn: 0, Inflight: 20,
> > pagedInMessages.size 0, enqueueCount: 211, dequeueCount: 207 |
> > org.apache.activemq.broker.region.Queue | ActiveMQ Broker[localhost]
> > Scheduler
> >
> > (It can be seen above that enqueued and dequeued difference is 4
> > and that is the number that is appearing in the pending column for that
> > queue however clicking on the queue shows no messages).
> >
> > Any help or insight is appreciated. Should I file a bug?
> >
> > --
> > View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-not-expiring-all-messages-and-leaving-them-stuck-in-queue-tp4654172.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-not-expiring-all-messages-and-leaving-them-stuck-in-queue-tp4654172p4654179.html
>  To unsubscribe from ActiveMQ not expiring all messages and leaving them
> stuck in queue?, click here<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4654172&code=Ynl0ZWZsaW5nZXJAZ21haWwuY29tfDQ2NTQxNzJ8LTExMDA3NjUyMzg=>
> .
> NAML<http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>


--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-not-expiring-all-messages-and-leaving-them-stuck-in-queue-tp4654172p4654180.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: ActiveMQ not expiring all messages and leaving them stuck in queue?

Posted by Gary Tully <ga...@gmail.com>.

Do you think you could create a test case that can reproduce this
behavior? If so, please open a jira issue.
I wonder if this is related to inflight=20, those messages are
dispatched and unacked by some consumer, so they may not expire in
that scenario.
Are you on 5.6, if not it would be good to see if you can reproduce
with 5.6 or even better 5.7-SNAPSHOT (which is trunk).

On 19 July 2012 14:56, ByteFlinger <by...@gmail.com> wrote:
> I am having a small issue with activemq.
>
> I have a queue which I send messages to with a certain TTL. When these
> messages expire, once the queue is read I expect all expired messages to be
> moved towards ActiveMQ.DLQ however what is happening is that most will do
> that however a few (Usually between 1 to 5) just sit in the queue and will
> not move.
>
> You can see that the queue count on the web console is showing some messages
> in that queue and you can see the same exact different between enqueued and
> dequeued messages. Strangely when clicking in the queue in the web console
> shows no messages whatsoever (An empty view) even though the count shows
> they are there (and I know they are there because they never appear in my
> logs which listen and logs all messages in ActiveMQ.DLQ).
>
> This is happening consistently. Does anybody know what is causing this? It
> feels to me that it is a bug in activemq however I cannot find any
> information on it when searching around the net and debug logs don't tell me
> much other than the line below:
>
> 2012-07-19 15:42:43,417 | DEBUG | Expiring messages .. |
> org.apache.activemq.broker.region.Queue | ActiveMQ Broker[localhost]
> Scheduler
> 2012-07-19 15:42:43,417 | DEBUG | SomeQueue toPageIn: 0, Inflight: 20,
> pagedInMessages.size 0, enqueueCount: 211, dequeueCount: 207 |
> org.apache.activemq.broker.region.Queue | ActiveMQ Broker[localhost]
> Scheduler
>
> (It can be seen above that enqueued and dequeued difference is 4
> and that is the number that is appearing in the pending column for that
> queue however clicking on the queue shows no messages).
>
> Any help or insight is appreciated. Should I file a bug?
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-not-expiring-all-messages-and-leaving-them-stuck-in-queue-tp4654172.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



-- 
http://fusesource.com
http://blog.garytully.com