Help on Message Group

[Don Santillan <do...@gmail.com>]

Hello,

I am trying to create an application that would use a message queue that 
is consumed by multiple consumers running concurrently but would still 
maintain the order of message processing by a certain ID.

For example, I have a set of messages added to a queue:
ID1-M1
ID1-M2
ID2-M1
ID3-M1
ID2-M2
ID3-M2

I need the Messages coming from ID1 get consumed by Consumer1, ID2 to 
Consumer2, and ID3 to Consumer3, that maintains order such as Consumer1 
will process ID1-M1 and then ID1-M2, Consumer 2 process ID2-M1 and then 
ID2-M2, and Consumer3 process ID3-M1 and then ID3-M2.

I also like to have the Consumers come from a ConsumerPool so that if a 
consumer is done processing with the messages, it will go back to the 
pool and become available for other Messages owned by another ID. I also 
like to have the pool a behavior that would create a new consumer in the 
pool if a consumer dies so that a fix number of consumer is available 
from the pool.

I saw in the documentation that the Message Group, I think, fits into 
the requirement perfectly. Process messages asynchronous, maintain 
message order, and failover feature.

Unfortunately, I was not able to understand completely how to do it. I 
understand that the producer needs to set a property "JMSXGroupID" to 
identify which group the message will get associated and consumed by a 
single consumer. What I don't understand is how the broker will select 
which consumer to pass the message to? Even more, where will the 
consumers come from? Do I need to create, for example, 20 consumers and 
subscribe them all to the broker? What if the consumer one by one dies, 
can the broker automatically recreate all 20 consumers?

It would be greatly appreciated if anybody can write a simple code that 
would show how consumers get subscribed to the broker and have an 
explanation that my requirement would really fit into Message Group. If 
this is not possible through Message Group, any suggestions on how to 
implement this?

Thanks in advance!

-don

Re: Help on Message Group

[Don Santillan <do...@gmail.com>]

James Strachan wrote:
> On 2 August 2010 18:49, Don Santillan <do...@gmail.com> wrote:
>   
>> Hello,
>>
>> I am trying to create an application that would use a message queue that is
>> consumed by multiple consumers running concurrently but would still maintain
>> the order of message processing by a certain ID.
>>
>> For example, I have a set of messages added to a queue:
>> ID1-M1
>> ID1-M2
>> ID2-M1
>> ID3-M1
>> ID2-M2
>> ID3-M2
>>
>> I need the Messages coming from ID1 get consumed by Consumer1, ID2 to
>> Consumer2, and ID3 to Consumer3, that maintains order such as Consumer1 will
>> process ID1-M1 and then ID1-M2, Consumer 2 process ID2-M1 and then ID2-M2,
>> and Consumer3 process ID3-M1 and then ID3-M2.
>>
>> I also like to have the Consumers come from a ConsumerPool so that if a
>> consumer is done processing with the messages, it will go back to the pool
>> and become available for other Messages owned by another ID. I also like to
>> have the pool a behavior that would create a new consumer in the pool if a
>> consumer dies so that a fix number of consumer is available from the pool.
>>
>> I saw in the documentation that the Message Group, I think, fits into the
>> requirement perfectly. Process messages asynchronous, maintain message
>> order, and failover feature.
>>
>> Unfortunately, I was not able to understand completely how to do it. I
>> understand that the producer needs to set a property "JMSXGroupID" to
>> identify which group the message will get associated and consumed by a
>> single consumer. What I don't understand is how the broker will select which
>> consumer to pass the message to? Even more, where will the consumers come
>> from? Do I need to create, for example, 20 consumers and subscribe them all
>> to the broker? What if the consumer one by one dies, can the broker
>> automatically recreate all 20 consumers?
>>
>> It would be greatly appreciated if anybody can write a simple code that
>> would show how consumers get subscribed to the broker and have an
>> explanation that my requirement would really fit into Message Group. If this
>> is not possible through Message Group, any suggestions on how to implement
>> this?
>>     
>
> The broker chooses a consumer on demand to fill a hash bucket. As
> consumers die new consumers get chosen. So all you need to do is add
> message group IDs to your message (as specific as possible to use the
> biggest possible number of hash buckets) and create a number of
> consumers. There's no need to manually assign consumers to individual
> message group IDs or anything like that.
>
> Sometimes prefetch can be set high which can reduce the amount of
> concurrent consumers used; if you are only sending a small number of
> messages you might find that only a small number of consumers are
> actually used.
> http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html
> http://activemq.apache.org/what-is-the-prefetch-limit-for.html
>
> so you might want to set the prefetch to say 1 if you've only a
> relatively small number of messages to ensure a good spread across
> your consumers.
>
>   
Thanks for replying James, now I got the point. I understand now that 
all consumers that I created that is associated to the queue will get 
passed depending on the first group id it has received.

More questions though. How does a consumer die? Can you site an example? 
If I have 5 consumers created and associated to a queue, and if 5 of 
them dies, does it mean the broker won't get any consumer to process the 
pending messages anymore?

Another scenario, if the queue has 10 messages that belongs to different 
groups, and there are only 5 consumers associated to the queue, what 
will happen to the remaining 5 messages in the queue?

Re: Help on Message Group

[James Strachan <ja...@gmail.com>]

On 2 August 2010 18:49, Don Santillan <do...@gmail.com> wrote:
> Hello,
>
> I am trying to create an application that would use a message queue that is
> consumed by multiple consumers running concurrently but would still maintain
> the order of message processing by a certain ID.
>
> For example, I have a set of messages added to a queue:
> ID1-M1
> ID1-M2
> ID2-M1
> ID3-M1
> ID2-M2
> ID3-M2
>
> I need the Messages coming from ID1 get consumed by Consumer1, ID2 to
> Consumer2, and ID3 to Consumer3, that maintains order such as Consumer1 will
> process ID1-M1 and then ID1-M2, Consumer 2 process ID2-M1 and then ID2-M2,
> and Consumer3 process ID3-M1 and then ID3-M2.
>
> I also like to have the Consumers come from a ConsumerPool so that if a
> consumer is done processing with the messages, it will go back to the pool
> and become available for other Messages owned by another ID. I also like to
> have the pool a behavior that would create a new consumer in the pool if a
> consumer dies so that a fix number of consumer is available from the pool.
>
> I saw in the documentation that the Message Group, I think, fits into the
> requirement perfectly. Process messages asynchronous, maintain message
> order, and failover feature.
>
> Unfortunately, I was not able to understand completely how to do it. I
> understand that the producer needs to set a property "JMSXGroupID" to
> identify which group the message will get associated and consumed by a
> single consumer. What I don't understand is how the broker will select which
> consumer to pass the message to? Even more, where will the consumers come
> from? Do I need to create, for example, 20 consumers and subscribe them all
> to the broker? What if the consumer one by one dies, can the broker
> automatically recreate all 20 consumers?
>
> It would be greatly appreciated if anybody can write a simple code that
> would show how consumers get subscribed to the broker and have an
> explanation that my requirement would really fit into Message Group. If this
> is not possible through Message Group, any suggestions on how to implement
> this?

The broker chooses a consumer on demand to fill a hash bucket. As
consumers die new consumers get chosen. So all you need to do is add
message group IDs to your message (as specific as possible to use the
biggest possible number of hash buckets) and create a number of
consumers. There's no need to manually assign consumers to individual
message group IDs or anything like that.

Sometimes prefetch can be set high which can reduce the amount of
concurrent consumers used; if you are only sending a small number of
messages you might find that only a small number of consumers are
actually used.
http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html
http://activemq.apache.org/what-is-the-prefetch-limit-for.html

so you might want to set the prefetch to say 1 if you've only a
relatively small number of messages to ensure a good spread across
your consumers.

-- 
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://fusesource.com/