Re: Artemis: JMSXGroupID / Message groups: Large number of groups

[Martyn Taylor <mt...@redhat.com>]

Hi Marco,

Please see comments inline.

Regards
Martyn

On 08/06/15 20:48, Marco wrote:
> Hi all,
>
> first, thanks for the release of v1.0.0 of Artemis!
>
> I'm currently experimenting with the use of message groups to quickly
> distribute messages to workers while keeping order on items belonging
> together. In order to avoid artificial grouping/sharding it would be perfect
> for my use case if the UUID of an object could be used as the message group
> ID (to keep the order of modifications to this object).
>
> As mentioned e.g.  here
> <http://scott.cranton.com/2010/09/activemq-message-groups.html>  , there
> seems to be a need to reconfigure the "classic" ActiveMQ to use a different
> "messageGroupMapFactory" in order to use more than 1024 message groups. Of
> course, in theory, the UUID maps could be much larger than my memory can
> possibly hold, but if the implementation would throw away message group
> assignments if the map is full (i.e. roll over) while there is currently no
> message for this group ID, then this should work fine. The next incoming
> UUID that was thrown away before would just be reassigned to one of the
> consumers with the "first for this consumer" flag set.
>
> My questions are:
> - Does Artemis use the same default map as "classic" ActiveMQ, i.e. is there
> a limitation to 1024 group IDs?
We do not limit the number of groups IDs.
> - If yes, how can it be reconfigured to allow for more values?
> - If not, how is Artemis currently designed to handle this?
Each Queue has associated with it, a mapping from groups to consumer.  
We do a lookup to decide which consumer to forward to when the message 
is routed.
> - Is this use case feasible / Are there any experiences with such a
> grouping?
One thing to note with the approach you outlined above.  The Group ID on 
a Queue exists for the entire life time of the consumer that is 
associated with it.  In other words, Group IDs will not be removed from 
a Queues internal mapping until the consumer is closed.  If in your use 
case above, you have long lived consumers and are creating new groups 
regularly, the group -> consumer map could potentially grow quite 
large.  You can't explicitly remove groups from a queue, but you can get 
around this by setting a Local Group Handler (example below), which will 
clean up stale groups (Groups that have not received a message for a 
certain period of time).

For information on setting up GroupHandlers in a clustered environment 
see: https://activemq.apache.org/artemis/docs/1.0.0/message-grouping.html

GroupHandler config, (add to broker.xml):

|<grouping-handler name="my-grouping-handler">
    <type>LOCAL</type>
    <address>Foo</address>  <!--||This is a required field but is only used in a clustered topology, you can set this to anything for now.  The Group Handler is global, and works on all addresses.  -->
    <group-timeout>60000</group-timeout> <!-- The amount of time in ms before a group goes stale -->
</grouping-handler>|

>
> Thanks in advance,
> Marco
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Artemis-JMSXGroupID-Message-groups-Large-number-of-groups-tp4697436.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.