You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Arjen van der Meijden <ac...@tweakers.net> on 2009/06/08 15:40:03 UTC

Consumers newer than messages don't receive messages from queues

Hello list,

In a somewhat complicated set of consumption-processing, I have three 
queues. One to indicate a new job, which is consumed by a single 
long-running consumer that sends messages to a second queue.
That second queue is consumed by four consumers that kill themselves 
after having processed a (few) message(s). After each message from the 
second queue is processed, they're send to a third queue which has two 
similar consumers.
As soon as such a consumer exits, a control process starts a new 
consumer, to have a fixed amount of consumers.

The jobs for the initial queue are issued in batches, so there can be a 
burst of about 150 messages, resulting in a burst in the second queue.

As soon as those second-queue-consumers exit however, they don't consume 
much messages. Most of the time it seems that only one is actually doing 
any work and the other three are idle, even if the queue has more than 
100 messages. The same holds for the third-queue-consumers.

It seems to have to do with the fact that those consumers actually 
subscribe to the queue after the messages have been produced. Obviously, 
I want them to start consuming a message as soon as they connect.

All message production and consumption is via Stomp in PHP and the 
ActiveMQ is a 5.2.0-version with mostly a default configuration. I have 
tried to see if subscribing using the headers 'activemq.retroactive' and 
'activemq.dispatchAsync' set to 'true' make any difference, but that 
doesn't seem to do anything.

Anyone knows what's going on and how to fix it?

Best regards,

Arjen


Re: Consumers newer than messages don't receive messages from queues

Posted by Arjen van der Meijden <ac...@tweakers.net>.
On 8-6-2009 16:25 Aleksandar Ivanisevic wrote:
> Arjen van der Meijden
> <ac...@tweakers.net> writes:
> 
>> As soon as those second-queue-consumers exit however, they don't
>> consume much messages. Most of the time it seems that only one is
>> actually doing any work and the other three are idle, even if the
>> queue has more than 100 messages. The same holds for the
>> third-queue-consumers.
> 
> How sure are you that you are not subscribing exclusively?

Well, since you need to do (and/or configure) that explicitly. Very 
sure. But as said in my follow-up mail, it seems to have been the 
prefetchSize.

Best regards,

Arjen

Re: Consumers newer than messages don't receive messages from queues

Posted by Aleksandar Ivanisevic <al...@ivanisevic.de>.
Arjen van der Meijden
<ac...@tweakers.net> writes:

> As soon as those second-queue-consumers exit however, they don't
> consume much messages. Most of the time it seems that only one is
> actually doing any work and the other three are idle, even if the
> queue has more than 100 messages. The same holds for the
> third-queue-consumers.

How sure are you that you are not subscribing exclusively?



Re: Consumers newer than messages don't receive messages from queues

Posted by Arjen van der Meijden <ac...@tweakers.net>.
Well, I seem to have fixed it... I completely overlooked that the 
default for the prefetch is actually larger than 1. Setting it to 1 
fixes the problem. Still it seems to be a bit of a waste that several 
newly connected consumers can be idle while because a "old" one that is 
processing, is actually "locking" the other messages.

Best regards,

Arjen

On 8-6-2009 15:40, Arjen van der Meijden wrote:
> Hello list,
> 
> In a somewhat complicated set of consumption-processing, I have three 
> queues. One to indicate a new job, which is consumed by a single 
> long-running consumer that sends messages to a second queue.
> That second queue is consumed by four consumers that kill themselves 
> after having processed a (few) message(s). After each message from the 
> second queue is processed, they're send to a third queue which has two 
> similar consumers.
> As soon as such a consumer exits, a control process starts a new 
> consumer, to have a fixed amount of consumers.
> 
> The jobs for the initial queue are issued in batches, so there can be a 
> burst of about 150 messages, resulting in a burst in the second queue.
> 
> As soon as those second-queue-consumers exit however, they don't consume 
> much messages. Most of the time it seems that only one is actually doing 
> any work and the other three are idle, even if the queue has more than 
> 100 messages. The same holds for the third-queue-consumers.
> 
> It seems to have to do with the fact that those consumers actually 
> subscribe to the queue after the messages have been produced. Obviously, 
> I want them to start consuming a message as soon as they connect.
> 
> All message production and consumption is via Stomp in PHP and the 
> ActiveMQ is a 5.2.0-version with mostly a default configuration. I have 
> tried to see if subscribing using the headers 'activemq.retroactive' and 
> 'activemq.dispatchAsync' set to 'true' make any difference, but that 
> doesn't seem to do anything.
> 
> Anyone knows what's going on and how to fix it?
> 
> Best regards,
> 
> Arjen
> 
>