You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by mclaudio76 <cl...@limontainformatica.it> on 2015/01/16 08:58:26 UTC
Messages are kept in Pending queue for durable topic subscribers.
Hello everyone,
I have some durable topic subscribers of a certain topic ABC. When a
subscriber goes offline, from ActiveMQ console I can see that messages
routed to topic ABC are listed - correctly - even in ABC:subscriber's
pending queue. The problem is that many messages aren't sent to the
subscriber when it goes online again. These messages aren't expired because
I set a quite high time-to-live. All subscribers use AUTO_ACKNOWLEDGE mode
to notify the broker a message has been received.
Can anyone help me ? What may I verify ?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are kept in Pending queue for durable topic
subscribers.
Posted by mclaudio76 <cl...@limontainformatica.it>.
artnaseef wrote
> OK, so first off, be cautious of the meaning of the broker statistics.
> Negative consumer counts is definitely a concern and worthy of a bug
> report (hopefully with a unit test that replicates the problem - or at
> least simple instructions to reproduce it). But some of the other
> statistics can be misleading - especially enqueue and dequeue counts on
> Topics.
Ok, but I'm thinking more about my configuration issue than a bug in
ActiveMQ.
artnaseef wrote
> That temporary advisory is an attempt to optimize the case of a client
> producing messages to a temporary destination after that destination has
> been removed. Every connection consumes from that advisory by default
> unless the "jms.watchTopicAdvisories" flag is set on the connection URI;
> like this:
> tcp://localhost:61616?jms.watchTopicAdvisories=false
> However, that subscription should never be durable.
Ok, now I've added watchTopicAdivsories=false, let's see what happen after
that. I would expect no more AdvisoryTopic being created, and no more
durable subscriptions associated both with the "real" topic and to the
AdvisoryTopic.
artnaseef wrote
> Back to the answer of how missing messages are detected. Increasing
> "Pending Queue" counts shows messages being stored - not lost. How is the
> loss detected?
The loss isn't detected, it's presumed:a message stored in the Pending Queue
and never removed from there makes me think that such message will never be
delivered to the client.
artnaseef wrote
> On the topic of disk space use: keep in mind that a single message can
> hold an entire KahaDB data file (typically 32mb), leading to large amounts
> of unused KahaDB space. Durable subscribers are a risk for causing a
> problem of this type because they are designed to allow consumers to go
> offline - which creates a "slow consumer" scenario (really a no-consumer
> scenario). And when they are offline, their messages sit unconsumed in
> the data files while other messages continue to flow through the system.
Ok, but messages have an expiration time. I thought that when a message
expires, it is not delivered, while it is put in DLQ queue. Is it right to
presume that KahaDB is freed when a message is expired or not ? About
durable subscribers: as mentioned before, I'm using JMS to notify a Java
based client application that an event occurred on a remote server. Those
clients are used by human users - no application to application scenario.
It's quite normal that an user launches a long-lasting processing job, goes
home, and next day when he or she arrives should be notified that his or her
work has been completed. Without durable subscriptions, this is not possible
AFAIK.
Hope this helps.
I really thank you for your time and precious explanation. I'm sorry to ask
questions that maybe are silly, but I'm really a newbe on ActiveMQ.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015p4690702.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are kept in Pending queue for durable topic
subscribers.
Posted by artnaseef <ar...@artnaseef.com>.
OK, so first off, be cautious of the meaning of the broker statistics.
Negative consumer counts is definitely a concern and worthy of a bug report
(hopefully with a unit test that replicates the problem - or at least simple
instructions to reproduce it). But some of the other statistics can be
misleading - especially enqueue and dequeue counts on Topics.
That temporary advisory is an attempt to optimize the case of a client
producing messages to a temporary destination after that destination has
been removed. Every connection consumes from that advisory by default
unless the "jms.watchTopicAdvisories" flag is set on the connection URI;
like this:
tcp://localhost:61616?jms.watchTopicAdvisories=false
However, that subscription should never be durable.
Back to the answer of how missing messages are detected. Increasing
"Pending Queue" counts shows messages being stored - not lost. How is the
loss detected?
On the topic of disk space use: keep in mind that a single message can hold
an entire KahaDB data file (typically 32mb), leading to large amounts of
unused KahaDB space. Durable subscribers are a risk for causing a problem
of this type because they are designed to allow consumers to go offline -
which creates a "slow consumer" scenario (really a no-consumer scenario).
And when they are offline, their messages sit unconsumed in the data files
while other messages continue to flow through the system.
Hope this helps.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015p4690679.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are kept in Pending queue for durable topic
subscribers.
Posted by mclaudio76 <cl...@limontainformatica.it>.
Hi artnaseef,
messages are meant to be simply notification of completion of remote jobs
sent to the client. So, loss of messages hasn't much impact on the normal
activity of the clients. This loss is detected by the fact that durable
subscribers (which go off and on line) have in their "Pending queue" quite
a good number of messages, and this number grows up. More over, storage
usage is growing up too - i can list a number of kahadb log files of about
32 Mb each.
As previously posted, I supposed that presence of Advisory.Temp topics might
create the issue, so I disabled them. The problem is that after a while , a
ActiveMQ.Advisory.TempQueue,ActiveMQ.Advisory.TempTopic topic has been
created. Now it has -95 (yes, a negative number !!!) consumers registered.
What the hell is it ? If I click on a durable subscriber, via web console,
I see that it's registered to the persistent topic (and that's ok) AND to
this TempTopic I cannot understand what it may be
<http://activemq.2283324.n4.nabble.com/file/n4690591/topics.png> :
Thanks again for your support.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015p4690591.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are kept in Pending queue for durable topic
subscribers.
Posted by artnaseef <ar...@artnaseef.com>.
Another thing to look at here: how is the message "loss" detected? Is it by
looking at broker statistics? Or, is there a problem at the application
level that makes it clear messages are missing?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015p4690565.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are kept in Pending queue for durable topic subscribers.
Posted by Gary Tully <ga...@gmail.com>.
by default the broker retains the sub in memory for an offline durable
sub, that can be disabled with keepDurableSubsActive=false broker
attribute. That may sort out the behaviour because the sub will be
replayed from the store.
One other thought, there was an issue with the default advisory
consumer on a jms connection (tracking temps) using a sync consumer,
so that if it blocked for some reason, that could filter back to the
broker. That could impact advisory messages.
jms.watchTopicAdvisories=false would avoid that.
But as ever, a test case that can reproduce (on trunk) is the best way
to find the root cause. good luck.
On 22 January 2015 at 14:41, mclaudio76
<cl...@limontainformatica.it> wrote:
> It's a standalone broker. Unfortunately, I cannot reproduce the problem in a
> test enviroment, mainly because I cannot generate all the traffic the
> production broker has to support. Anyway, within the limits of a test
> simulation, the presence or not of Advisory topics seemed not to influence
> the behaviour of the test broker.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015p4690273.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are kept in Pending queue for durable topic
subscribers.
Posted by mclaudio76 <cl...@limontainformatica.it>.
It's a standalone broker. Unfortunately, I cannot reproduce the problem in a
test enviroment, mainly because I cannot generate all the traffic the
production broker has to support. Anyway, within the limits of a test
simulation, the presence or not of Advisory topics seemed not to influence
the behaviour of the test broker.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015p4690273.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are kept in Pending queue for durable topic subscribers.
Posted by Tim Bain <tb...@alumni.duke.edu>.
That's strange. Is this a network of brokers or a standalone broker?
Also, can you reliably reproduce this (in a rest environment) just by
turning advisory messages back on? That test would confirm (or refute) the
link between advisory messages and the problem, and make it more likely
that someone will find the root cause and fix it.
On Jan 22, 2015 4:24 AM, "mclaudio76" <cl...@limontainformatica.it>
wrote:
> tbain98 wrote
> > So what happens when the subscriber comes back online? Are other
> messages
> > delivered, just not the ones that were sent while the subscriber was
> > offline? After the subscriber comes back online, do you still see those
> > messages pending in the JMX stats and/or the web console? In short,
> > you've
> > told us what doesn't happen, but not what does, which may be why no one
> > responded to try to help you; give us more information, and someone may
> > have a suggestion for you.
>
> Yeah, you're right... sorry for not have given more details about what
> happens. New income messages are correctly delivered to the subscriber when
> it comes back online. The problem *was* that there were quite a good number
> of pending messages that were never dequeued. I wrote *was* because I tried
> to turn off Advisory Messages, and since then this problem seems to have
> been disappeared.
> No idea how this change may affected ActiveMQ behaviour.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015p4690250.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Messages are kept in Pending queue for durable topic
subscribers.
Posted by mclaudio76 <cl...@limontainformatica.it>.
tbain98 wrote
> So what happens when the subscriber comes back online? Are other messages
> delivered, just not the ones that were sent while the subscriber was
> offline? After the subscriber comes back online, do you still see those
> messages pending in the JMX stats and/or the web console? In short,
> you've
> told us what doesn't happen, but not what does, which may be why no one
> responded to try to help you; give us more information, and someone may
> have a suggestion for you.
Yeah, you're right... sorry for not have given more details about what
happens. New income messages are correctly delivered to the subscriber when
it comes back online. The problem *was* that there were quite a good number
of pending messages that were never dequeued. I wrote *was* because I tried
to turn off Advisory Messages, and since then this problem seems to have
been disappeared.
No idea how this change may affected ActiveMQ behaviour.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015p4690250.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are kept in Pending queue for durable topic subscribers.
Posted by Tim Bain <tb...@alumni.duke.edu>.
So what happens when the subscriber comes back online? Are other messages
delivered, just not the ones that were sent while the subscriber was
offline? After the subscriber comes back online, do you still see those
messages pending in the JMX stats and/or the web console? In short, you've
told us what doesn't happen, but not what does, which may be why no one
responded to try to help you; give us more information, and someone may
have a suggestion for you.
Also, does this happen reproducibly in a non-production environment where
it would be safe to attach a debugger and step through code without
affecting production? If so, that may open more avenues for investigating
the issue than if we're stuck with examining log lines and JMX stats.
On Fri, Jan 16, 2015 at 12:58 AM, mclaudio76 <
claudio.mauri@limontainformatica.it> wrote:
> Hello everyone,
>
> I have some durable topic subscribers of a certain topic ABC. When a
> subscriber goes offline, from ActiveMQ console I can see that messages
> routed to topic ABC are listed - correctly - even in ABC:subscriber's
> pending queue. The problem is that many messages aren't sent to the
> subscriber when it goes online again. These messages aren't expired because
> I set a quite high time-to-live. All subscribers use AUTO_ACKNOWLEDGE mode
> to notify the broker a message has been received.
>
> Can anyone help me ? What may I verify ?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Messages-are-kept-in-Pending-queue-for-durable-topic-subscribers-tp4690015.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>