You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by jbar <ja...@gmail.com> on 2017/07/20 12:03:38 UTC
Messages are not discarded from queue when they're expired.
Hello,
Messages from my queue (httpTestConn_notifications) won't get discarded. On
a producer side, I've set a TTL of 2000 ms (with the setTimeToLive()
method).
Im using a 5.15 version of activemq (however I've tried to do that on 13.2
as well) . Here's a part of my activemq.xml:
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
advisorySupport="false" persistent="false">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue="*.prioritized" prioritizedMessages="true"
useCache="false" expireMessagesPeriod="0" queuePrefetch="1"
memoryLimit="10mb" producerFlowControl="true">
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
<policyEntry queue="httpTestConn_notifications"
expireMessagesPeriod="3000" memoryLimit="50mb" producerFlowControl="true">
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
</policyEntry>
<policyEntry queue=">" expireMessagesPeriod="3000" memoryLimit="50mb"
producerFlowControl="true">
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false" />
</deadLetterStrategy>
</policyEntry>
<policyEntry topic=">" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="100"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
When I disable the consumer to ensure that ttl will be reached, the message
just stays pending/enqueued forever, and I want it to disappear.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Messages-are-not-discarded-from-queue-when-they-re-expired-tp4728731.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are not discarded from queue when they're expired.
Posted by Tim Bain <tb...@alumni.duke.edu>.
On Fri, Jul 21, 2017 at 8:30 AM, jbar <ja...@gmail.com> wrote:
> >When you say the messages "are indeed purged," do you mean to say (in more
> >words, to make sure we're clear) "When I reconnect my consumer that has
> >been offline for longer than the TTL I set, all messages older than the
> TTL
> >are expired (as confirmed by an increase in the ExpiredCount JMX attribute
> >on the queue in question or by some similar mechanism)"?
>
> In case of my project, the consumer is a big message collecting system with
> gui, so I literally see that there are no new messages after reconnecting
> it
> (even though they were sent by the producer and enqueued before) in both
> cases, with and without given TTL. As soon as the consumer goes online, the
> pending messages counter starts decreasing, and the number of dequeued
> messages starts going up.
>
I would expect the ExpiredCount JMX counter to increase, not the
DequeueCount one. That implies that the messages are in fact being sent to
the consumer even though they're expired. If so, you'd see that in the JMX
stats (the InFlightCount would be non-zero). Do you see evidence that the
messages are being dispatched to the consumer in the case of a consumer
that was offline and only reconnects after all messages are expired?
> >What about your test procedure causes them to be clearly pending for
> >more than half a second? Are you saying you're publishing messages far
> >faster than your consumer can process them and therefore there's a large
> >backlog?
>
> Yes.
>
> >How are you determining that message expiration isn't happening?
>
> I just see 10k delivered messages on a consumer side.
>
Does "delivered" mean "successfully consumed by your application," or
something else? I want to make sure I'm not misinterpreting what you're
saying.
> >With this scenario, message expiration could be taking place on either the
> >broker or the consumer
>
> With my scenario messages are pending for more than 500 ms (it takes a
> dozen
> or so seconds to dequeue them) therefore I guess it should take place on
> broker.
>
I agree with that analysis, based on that information. While messages are
still being produced, it could be taking place in both locations (though
only if the consumer finished consuming the first message while messages
are still being produced), but 500 ms after the last message is produced,
expiration should happen in the broker for all messages except the ones in
the consumer's prefetch buffer. Since your prefetch is not set for this
topic, it will use the default value of 1000 per consumer, so I'd expect to
see ~1000 messages expired on the consumer side and ~9000 expired on the
broker.
BTW, your process is only using a single consumer on this queue, right?
You're not doing anything like running 10 consumers in parallel, or running
a threadpool of 10,000 threads that each process a single message at a
time, such that all of the messages could dispatched before they expire,
are you? I assume not, but just need to ask.
> >But I would expect that 1/2
> >second after your last message is produced, your consumer would receive no
> >further messages to process.
>
> That's excactly what I'm trying to achieve.
>
> >are the clocks synchronized on whatever machines are
> >running your producers, consumers, and brokers?
>
> In my tests the producer, consumer and broker are running on the same
> machine, therefore that's not the case since they're using the same system
> time.
>
OK, thanks for confirming.
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ActiveMQ-Messages-are-not-discarded-from-queue-
> when-they-re-expired-tp4728731p4728770.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Can you please use the queue browser page in the web console to view the
headers of a message *before it expires* so we can confirm that the
JMSExpiration header is in fact being set (and set to a correct value
that's greater than the creation time by the TTL value)? For this to work,
you might have to extend the TTL window, because with only half a second,
your odds of accessing the message via the web console are pretty slim. I
assume that you'll find that the value of that header is correct (meaning
that the problem is in the broker and/or the consumer rather than the
producer), but let's confirm that assumption so we can rule out part of the
potential problem space.
Tim
Re: Messages are not discarded from queue when they're expired.
Posted by jbar <ja...@gmail.com>.
>When you say the messages "are indeed purged," do you mean to say (in more
>words, to make sure we're clear) "When I reconnect my consumer that has
>been offline for longer than the TTL I set, all messages older than the TTL
>are expired (as confirmed by an increase in the ExpiredCount JMX attribute
>on the queue in question or by some similar mechanism)"?
In case of my project, the consumer is a big message collecting system with
gui, so I literally see that there are no new messages after reconnecting it
(even though they were sent by the producer and enqueued before) in both
cases, with and without given TTL. As soon as the consumer goes online, the
pending messages counter starts decreasing, and the number of dequeued
messages starts going up.
>What about your test procedure causes them to be clearly pending for
>more than half a second? Are you saying you're publishing messages far
>faster than your consumer can process them and therefore there's a large
>backlog?
Yes.
>How are you determining that message expiration isn't happening?
I just see 10k delivered messages on a consumer side.
>With this scenario, message expiration could be taking place on either the
>broker or the consumer
With my scenario messages are pending for more than 500 ms (it takes a dozen
or so seconds to dequeue them) therefore I guess it should take place on
broker.
>But I would expect that 1/2
>second after your last message is produced, your consumer would receive no
>further messages to process.
That's excactly what I'm trying to achieve.
>are the clocks synchronized on whatever machines are
>running your producers, consumers, and brokers?
In my tests the producer, consumer and broker are running on the same
machine, therefore that's not the case since they're using the same system
time.
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Messages-are-not-discarded-from-queue-when-they-re-expired-tp4728731p4728770.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are not discarded from queue when they're expired.
Posted by Tim Bain <tb...@alumni.duke.edu>.
When you say the messages "are indeed purged," do you mean to say (in more
words, to make sure we're clear) "When I reconnect my consumer that has
been offline for longer than the TTL I set, all messages older than the TTL
are expired (as confirmed by an increase in the ExpiredCount JMX attribute
on the queue in question or by some similar mechanism)"?
For your second test scenario (10000 messages, consumer connected all the
time), what about your test procedure causes them to be clearly pending for
more than half a second? Are you saying you're publishing messages far
faster than your consumer can process them and therefore there's a large
backlog? How are you determining that message expiration isn't happening?
With this scenario, message expiration could be taking place on either the
broker or the consumer, so to get a count of expired messages you have to
sum together the count from JMX for the broker plus the number of expired
messages listed in the consumer logs (which is only written every 1000
expired messages, if I remember correctly). But I would expect that 1/2
second after your last message is produced, your consumer would receive no
further messages to process.
One other question: are the clocks synchronized on whatever machines are
running your producers, consumers, and brokers? If not, that could explain
the behavior you're seeing.
Tim
On Fri, Jul 21, 2017 at 3:10 AM, jbar <ja...@gmail.com> wrote:
> Thank you for your response.
>
> The messages are indeed purged when I reconnect my customer, however I
> observed the same behaviour testing one that doesn't set any TTL. When
> trying the mechanism "in action" with 10 000 messages and their TTL = 500
> ms
> (and consumer connected all the time), all of them get dequeued and
> delivered to the consumer, despite clearly being pending for more than only
> 0,5 s.
> Am I missing something? Maybe there're some files other than activemq.xml
> that I should take a look at?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ActiveMQ-Messages-are-not-discarded-from-queue-
> when-they-re-expired-tp4728731p4728760.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Messages are not discarded from queue when they're expired.
Posted by jbar <ja...@gmail.com>.
Thank you for your response.
The messages are indeed purged when I reconnect my customer, however I
observed the same behaviour testing one that doesn't set any TTL. When
trying the mechanism "in action" with 10 000 messages and their TTL = 500 ms
(and consumer connected all the time), all of them get dequeued and
delivered to the consumer, despite clearly being pending for more than only
0,5 s.
Am I missing something? Maybe there're some files other than activemq.xml
that I should take a look at?
--
View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Messages-are-not-discarded-from-queue-when-they-re-expired-tp4728731p4728760.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Messages are not discarded from queue when they're expired.
Posted by Tim Bain <tb...@alumni.duke.edu>.
In my experience, message expiration is only checked on the broker when
preparing to dispatch a message and in the consumer when preparing to
consume it. So what you're describing sounds like expected behavior, but
I'd expect all those messages to be expired when your consumer reconnects.
On Jul 20, 2017 7:05 AM, "jbar" <ja...@gmail.com> wrote:
> Hello,
>
> Messages from my queue (httpTestConn_notifications) won't get discarded. On
> a producer side, I've set a TTL of 2000 ms (with the setTimeToLive()
> method).
> Im using a 5.15 version of activemq (however I've tried to do that on 13.2
> as well) . Here's a part of my activemq.xml:
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
> advisorySupport="false" persistent="false">
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue="*.prioritized"
> prioritizedMessages="true"
> useCache="false" expireMessagesPeriod="0" queuePrefetch="1"
> memoryLimit="10mb" producerFlowControl="true">
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> <policyEntry queue="httpTestConn_notifications"
> expireMessagesPeriod="3000" memoryLimit="50mb" producerFlowControl="true">
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false" />
> </deadLetterStrategy>
> </policyEntry>
> <policyEntry queue=">" expireMessagesPeriod="3000"
> memoryLimit="50mb"
> producerFlowControl="true">
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false" />
> </deadLetterStrategy>
> </policyEntry>
>
> <policyEntry topic=">" >
>
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="100"/>
> </pendingMessageLimitStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
>
> When I disable the consumer to ensure that ttl will be reached, the message
> just stays pending/enqueued forever, and I want it to disappear.
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Messages-are-not-discarded-from-queue-when-
> they-re-expired-tp4728731.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>