1 Queue with 2 Consumers - message delivery order?

[Praveen M <le...@gmail.com>]

Hi,

I ran the following test

1) I created 1 Queue
2) Registered 2 consumers to the queue
3) Enqueued 10 messages to the Queue. [ The first enqueued message is long
running. I simulated such that the first message on consumption takes about
50 seconds to be processed]
4) Once the enqueue is committed, the 2 consumers each pick a message.
5) The 1st consumer that got the long running message works on it for a long
time while the second consumer that got the second message keeps processing
and going to the next message, but  only goes as far until it processes 5 of
the 10 messages enqueued. Then the 2nd consumer gives up processing.
6) When the 1st consumer with the  long running message completes, it then
ends up processing the remaining messages and my test completes.

So it seems like the two consumers were trying to take a fair share of
messages that they were processing immaterial of the time it takes to
process individual messages. Enqueued message = 10, Consumer 1 share of 5
messages were processed by it, and Consumer 2's share of 5 messages were
processed by it.


This is kinda against the behavior that I'd like to see. The desired
behavior in my case is that of each consumer keeps going on if it's done and
has other messages to process.

In the above test, I'd expect as consumer 1 is working on the long message,
the second consumer should work its way through all the remaining messages.

Is there some config that I'm missing that could cause this effect?? Any
advice on tackling this will be great.

Also, Can someone please explain in what order are messages delivered to the
consumers in the following cases?

Case 1)
 There is a single Queue with more than 1 message in it and multiple
consumers registered to it.

Case 2)
There are multiple queues each with more than 1 message in it, and has
multiple consumers registered to it.



Thank you,
-- 
-Praveen

Re: 1 Queue with 2 Consumers - message delivery order?

[Robbie Gemmell <ro...@gmail.com>]

As Jakub mentioned, this '5 messages each' behaviour is the result of
prefetch (and your consumers being present before you started
publishing). Because the consumers had space in their prefetch buffer,
the messages were basically round-robin delivered to the clients as
they were enqueued.

The delivery mechanism is quite complicated as there are 3 ways
messages get dispatched to the client, depending on the state of each
client when the message is enqueued and the prefetch configuration.
The queue makes a basic attempt immediately to round-robin deliver
incoming messages to the consumers if there is mroe than 1 present on
a queue. If the message cant be delivered to a consumer immedaitely
upon enqueue then an asynchronous delivery process is started to
attempt message delivery, which again makes a basic attempt to
round-robin the messages on the queue to the available consumers. A
third mechanism kicks in when a client newly connects or goes from not
having room in its prefetch buffer to having room, in which a
different asynchronous delivery mechnism attempts to deliver messages
to that specific consumer. Different queues operate entirely
independently in delivery terms, other than use of a shared thread
pool for the asynchronous delivery processing.

Robbie

On 26 October 2011 02:02, Praveen M <le...@gmail.com> wrote:
> Hi,
>
> I ran the following test
>
> 1) I created 1 Queue
> 2) Registered 2 consumers to the queue
> 3) Enqueued 10 messages to the Queue. [ The first enqueued message is long
> running. I simulated such that the first message on consumption takes about
> 50 seconds to be processed]
> 4) Once the enqueue is committed, the 2 consumers each pick a message.
> 5) The 1st consumer that got the long running message works on it for a long
> time while the second consumer that got the second message keeps processing
> and going to the next message, but  only goes as far until it processes 5 of
> the 10 messages enqueued. Then the 2nd consumer gives up processing.
> 6) When the 1st consumer with the  long running message completes, it then
> ends up processing the remaining messages and my test completes.
>
> So it seems like the two consumers were trying to take a fair share of
> messages that they were processing immaterial of the time it takes to
> process individual messages. Enqueued message = 10, Consumer 1 share of 5
> messages were processed by it, and Consumer 2's share of 5 messages were
> processed by it.
>
>
> This is kinda against the behavior that I'd like to see. The desired
> behavior in my case is that of each consumer keeps going on if it's done and
> has other messages to process.
>
> In the above test, I'd expect as consumer 1 is working on the long message,
> the second consumer should work its way through all the remaining messages.
>
> Is there some config that I'm missing that could cause this effect?? Any
> advice on tackling this will be great.
>
> Also, Can someone please explain in what order are messages delivered to the
> consumers in the following cases?
>
> Case 1)
>  There is a single Queue with more than 1 message in it and multiple
> consumers registered to it.
>
> Case 2)
> There are multiple queues each with more than 1 message in it, and has
> multiple consumers registered to it.
>
>
>
> Thank you,
> --
> -Praveen
>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

Re: 1 Queue with 2 Consumers - message delivery order?

[Jakub Scholz <ja...@scholz.cz>]

Hi Praveen,

Have you set the capacity / prefetch for the receivers to one message?
I believe the capacity defines how many messages can be "buffered" by
the client API in background while you are still processing the first
message. That may cause that both your clients receive 5 messages,
even when the processing in the first client takes a longer time.

Regards
Jakub

On Wed, Oct 26, 2011 at 03:02, Praveen M <le...@gmail.com> wrote:
> Hi,
>
> I ran the following test
>
> 1) I created 1 Queue
> 2) Registered 2 consumers to the queue
> 3) Enqueued 10 messages to the Queue. [ The first enqueued message is long
> running. I simulated such that the first message on consumption takes about
> 50 seconds to be processed]
> 4) Once the enqueue is committed, the 2 consumers each pick a message.
> 5) The 1st consumer that got the long running message works on it for a long
> time while the second consumer that got the second message keeps processing
> and going to the next message, but  only goes as far until it processes 5 of
> the 10 messages enqueued. Then the 2nd consumer gives up processing.
> 6) When the 1st consumer with the  long running message completes, it then
> ends up processing the remaining messages and my test completes.
>
> So it seems like the two consumers were trying to take a fair share of
> messages that they were processing immaterial of the time it takes to
> process individual messages. Enqueued message = 10, Consumer 1 share of 5
> messages were processed by it, and Consumer 2's share of 5 messages were
> processed by it.
>
>
> This is kinda against the behavior that I'd like to see. The desired
> behavior in my case is that of each consumer keeps going on if it's done and
> has other messages to process.
>
> In the above test, I'd expect as consumer 1 is working on the long message,
> the second consumer should work its way through all the remaining messages.
>
> Is there some config that I'm missing that could cause this effect?? Any
> advice on tackling this will be great.
>
> Also, Can someone please explain in what order are messages delivered to the
> consumers in the following cases?
>
> Case 1)
>  There is a single Queue with more than 1 message in it and multiple
> consumers registered to it.
>
> Case 2)
> There are multiple queues each with more than 1 message in it, and has
> multiple consumers registered to it.
>
>
>
> Thank you,
> --
> -Praveen
>

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org