You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by krishy <ca...@gmail.com> on 2015/01/15 21:07:03 UTC
Consumer queues of virtual topic and the web console
We are using the stock ActiveMQ 5.10.0 configuration (with minor
modifications for JMX). With this configuration we have run into a very
strange issue every now and then with messages not being dispatched from the
consumer queues of a virtual topic until we click 'browse' on the web
console of that queue! On clicking browse, a flood of message is received by
the consumers and then nothing. This problem, seemingly, pops up randomly
and is not rectified until we restart ActiveMQ. What could explain this
behaviour?
On restarting the ActiveMQ service, the message dispatch returns to normal
and stays that way for some time.
The destinationPolicy configuration is (from the vanilla 5.10 setup):
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
We use Apache Camel for routing with the ActiveMQComponent (and a
org.apache.activemq.pool.PooledConnectionFactory). The activemq client
version is 5.5 (is this a problem?).
Thanks for any and all pointers!
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer queues of virtual topic and the web console
Posted by Tim Bain <tb...@alumni.duke.edu>.
I see that Tim closed your JIRA as Not A Problem, but didn't explain why.
In case you weren't sure why this is expected behavior, here's an
explanation:
Messages get read from the broker's persistent store via a cursor (see
http://activemq.apache.org/message-cursors.html), which is assumed to
always have the next message any consumer might want. (This is sometimes a
bad assumption, but it's the design.) A cursor has a fixed size; once it's
full, a new message cannot be read into the cursor (and therefore delivered
to a consumer) until enough messages are read from the cursor to free up
space for the new message. If none of the existing messages in the cursor
are read, nothing happens.
In your case, you're intentionally creating messages that won't be
consumed. These messages build up in the cursor, and once it's full, none
of the new messages can get into the cursor so none of them can get
dispatched so none of them get consumed. And for that reason all messages
stop flowing.
Tim's suggestion to tweak the cursor size wouldn't solve this problem, it
would just delay the onset, in the same way that increasing your JVM heap
size when you have a memory leak only delays the problem rather than
solving it. The underlying issue is that you're doing something wrong:
you're publishing messages to a queue without ensuing that all of them will
be processed (consumed or discarded) in a timely manner. So either make
sure your consumers have selectors that match all messages or configure
something to discard messages if they aren't consumed in a timely manner
(JMSExpiration plus the DiscardingDLQPlugin is what we use).
Tim
On Feb 11, 2015 10:22 PM, "krishy" <ca...@gmail.com> wrote:
> I went ahead and created a ticket as well:
>
> https://issues.apache.org/jira/browse/AMQ-5582
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4691427.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Consumer queues of virtual topic and the web console
Posted by krishy <ca...@gmail.com>.
I went ahead and created a ticket as well:
https://issues.apache.org/jira/browse/AMQ-5582
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4691427.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer queues of virtual topic and the web console
Posted by krishy <ca...@gmail.com>.
tbain98 wrote
> And what's your expect behavior for the messages that don't match the
> filter for your consumer? What do you anticipate happening to them with
> this configuration?
Since the messages I send are non-persistent I would expect the
messages that do not match the selector to stay in the queue until
ActiveMQ is restarted. But messages that match the selector being
delayed is what confuses me.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690510.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer queues of virtual topic and the web console
Posted by Tim Bain <tb...@alumni.duke.edu>.
And what's your expect behavior for the messages that don't match the
filter for your consumer? What do you anticipate happening to them with
this configuration?
On Tue, Jan 27, 2015 at 3:15 PM, krishy <ca...@gmail.com> wrote:
> tbain98 wrote
> > OK, so I finally got around to looking at your producer and consumer code
> > in BitBucket. So it looks like you're intentionally only consuming half
> > the messages you produce; is that right? If so, to what end? You never
> > mentioned this particular characteristic of your test; any reason you
> > didn't?
>
> I am sorry, I should have been more clear about that. As I mentioned - "One
> sends messages to the virtual topic and another consumes from the queue of
> the virtual topic based on a selector." So, yes, the consumer does not care
> about all the messages sent to the virtualtopic/queue but is rather
> interested in only certain messages.
>
> The simplification of a scenario we use this is to pick messages from a
> stream of events. The events are pumped into a virtual topic and having
> multiple consumers subscribed to the corresponding queue allows us to run
> multiple instances of the consumer application (and thereby load balance
> the
> downstream processing).
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690491.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Consumer queues of virtual topic and the web console
Posted by krishy <ca...@gmail.com>.
tbain98 wrote
> OK, so I finally got around to looking at your producer and consumer code
> in BitBucket. So it looks like you're intentionally only consuming half
> the messages you produce; is that right? If so, to what end? You never
> mentioned this particular characteristic of your test; any reason you
> didn't?
I am sorry, I should have been more clear about that. As I mentioned - "One
sends messages to the virtual topic and another consumes from the queue of
the virtual topic based on a selector." So, yes, the consumer does not care
about all the messages sent to the virtualtopic/queue but is rather
interested in only certain messages.
The simplification of a scenario we use this is to pick messages from a
stream of events. The events are pumped into a virtual topic and having
multiple consumers subscribed to the corresponding queue allows us to run
multiple instances of the consumer application (and thereby load balance the
downstream processing).
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690491.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer queues of virtual topic and the web console
Posted by Tim Bain <tb...@alumni.duke.edu>.
OK, so I finally got around to looking at your producer and consumer code
in BitBucket. So it looks like you're intentionally only consuming half
the messages you produce; is that right? If so, to what end? You never
mentioned this particular characteristic of your test; any reason you
didn't?
On Jan 27, 2015 7:01 AM, "Tim Bain" <tb...@alumni.duke.edu> wrote:
> Since it sounds like you were able to create a simple setup to cause the
> issue, can you package it up, create a JIRA bug, and attach your test
> scenario to the JIRA? That way someone can step through it with a debugger
> and figure out what's going on.
> On Jan 23, 2015 10:40 AM, "krishy" <ca...@gmail.com> wrote:
>
>> I made the virtual topic selectorAware and the problem went away. All
>> messages are received by the consumer without much delays.
>>
>> I was able to reproduce this only with queues as well. That is the
>> messages
>> were sent to and consumed from a queue based on a selector. After a while
>> the consumer seemed to stop receiving messages. I
>>
>> Though my problem is solved, I am curious what is controlling this
>> behaviour. Why don't messages turn up from the queue after a while?
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690349.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
Re: Consumer queues of virtual topic and the web console
Posted by Tim Bain <tb...@alumni.duke.edu>.
Since it sounds like you were able to create a simple setup to cause the
issue, can you package it up, create a JIRA bug, and attach your test
scenario to the JIRA? That way someone can step through it with a debugger
and figure out what's going on.
On Jan 23, 2015 10:40 AM, "krishy" <ca...@gmail.com> wrote:
> I made the virtual topic selectorAware and the problem went away. All
> messages are received by the consumer without much delays.
>
> I was able to reproduce this only with queues as well. That is the messages
> were sent to and consumed from a queue based on a selector. After a while
> the consumer seemed to stop receiving messages. I
>
> Though my problem is solved, I am curious what is controlling this
> behaviour. Why don't messages turn up from the queue after a while?
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690349.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Consumer queues of virtual topic and the web console
Posted by krishy <ca...@gmail.com>.
I made the virtual topic selectorAware and the problem went away. All
messages are received by the consumer without much delays.
I was able to reproduce this only with queues as well. That is the messages
were sent to and consumed from a queue based on a selector. After a while
the consumer seemed to stop receiving messages. I
Though my problem is solved, I am curious what is controlling this
behaviour. Why don't messages turn up from the queue after a while?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690349.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer queues of virtual topic and the web console
Posted by krishy <ca...@gmail.com>.
I am able to reproduce this consistently. Messages don't arrive at the
consumer of the virtual topic queue.
I have a simple test case in this maven project:
https://bitbucket.org/calvinkrishy/camel-jms-test
One should be able to run it with mvn camel:run. The test uses 5.10.0 as an
embedded broker. There are two Camel routes. One sends messages to the
virtual topic and another consumes from the queue of the virtual topic based
on a selector.
These are the patterns I observe:
- Initially message show up at the consumer almost immediately.
- By the 350-400th message there is a marked slow-down. Messages arrive in
batches at the consumer.
- After around the 700th message being sent, nothing comes to the consumer.
I have also tried it with stock ActiveMQ 5.10.0 installation (i.e. stand
alone broker, default install with no changes) and can still see the
problem. Even if I completely stop the program and start it again nothing
arrives at the consumer. Only after I restart the broker does the behavior
return to normal (and that too for a short time).
Any pointers?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690064.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Consumer queues of virtual topic and the web console
Posted by artnaseef <ar...@artnaseef.com>.
Does this happen without the JMX modifications?
That does sound odd. One question comes to mind - does the queue actually
exist in the broker at the time of the message dispatch? Check the
consumers - make sure they are active at the time the message is dispatched,
and that the broker sees the consumer and the queue.
The reason the browser can change this operation is that clicking on the
link to browse the queue will create it. That's one "oddity" of virtual
topics - that the messages are put in the queues based on the presence of
the queue at the time of message dispatch to the Topic.
If that is the cause, it is possible to force creation of the consumer
queues at broker startup through the activemq.xml file.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690004.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.