Message Group rebalancing on new consumers

[Mark Johnson <sp...@gmail.com>]

Hi,

We have Apache Camel routes consuming messages and updating a database. Due
to the nature of the business, there will be multiple messages in the queue
that need to update the same row in the db tables. This results in a lot of
db transaction rollbacks as the number of consumers increases.

We have proved to ourselves that using Message Groups with part of the db
table primary key as the JMSXGroupID resolves the rollback. There will also
be considerably more groups than consumers, so I have no concerns about
consumers being idle because all groups are already serviced.

We have also proved that we can horizontally scale the Camel deployment
based on queue depth.

What we can't work out is how to put the two solutions together, as whilst
Message Group will rebalance when a consumer disconnects, it does not
rebalance on new consumers, according to the documentation.

The documentation suggests that sending in a message with a -1 SeqNr will
force the group to be treated as new, however at the point where we are
scaling up there will already be thousands of messages on the queue.

Does the reset happen when the broker receives the message or when the
message gets to the head of the queue? If the latter, could I use message
priority to force it to the head?

If those conditions are met then I thinking that the Camel routes could
process 1k msgs then send in a -1 reset message for their current GroupID
to cause it to be rebalanced. Would also need to set prefetch=1 to ensure
that the client does not have messages buffered, although even if we didn't
do that, this pattern should reduce the number of rollbacks significantly.

Is there a better way for groups to be moved to new consumers?

Any suggestions gratefully received.

Mark Johnson

Re: Message Group rebalancing on new consumers

[Mark Johnson <sp...@gmail.com>]

Apologies as of course I should have included the version.

AMQ Classic 5.15.1.

We have done some internal benchmarking comparing with AMQ Artemis 2.24.0,
specifically on total sustainable throughput, but so far have found no
compelling reason to switch away from Classic.

On Sat, 26 Nov 2022 at 18:57, Justin Bertram <jb...@apache.org> wrote:

> What version of ActiveMQ are you using?
>
>
> Justin
>
> On Fri, Nov 25, 2022 at 4:20 AM Mark Johnson <sp...@gmail.com>
> wrote:
>
> > Hi,
> >
> > We have Apache Camel routes consuming messages and updating a database.
> Due
> > to the nature of the business, there will be multiple messages in the
> queue
> > that need to update the same row in the db tables. This results in a lot
> of
> > db transaction rollbacks as the number of consumers increases.
> >
> > We have proved to ourselves that using Message Groups with part of the db
> > table primary key as the JMSXGroupID resolves the rollback. There will
> also
> > be considerably more groups than consumers, so I have no concerns about
> > consumers being idle because all groups are already serviced.
> >
> > We have also proved that we can horizontally scale the Camel deployment
> > based on queue depth.
> >
> > What we can't work out is how to put the two solutions together, as
> whilst
> > Message Group will rebalance when a consumer disconnects, it does not
> > rebalance on new consumers, according to the documentation.
> >
> > The documentation suggests that sending in a message with a -1 SeqNr will
> > force the group to be treated as new, however at the point where we are
> > scaling up there will already be thousands of messages on the queue.
> >
> > Does the reset happen when the broker receives the message or when the
> > message gets to the head of the queue? If the latter, could I use message
> > priority to force it to the head?
> >
> > If those conditions are met then I thinking that the Camel routes could
> > process 1k msgs then send in a -1 reset message for their current GroupID
> > to cause it to be rebalanced. Would also need to set prefetch=1 to ensure
> > that the client does not have messages buffered, although even if we
> didn't
> > do that, this pattern should reduce the number of rollbacks
> significantly.
> >
> > Is there a better way for groups to be moved to new consumers?
> >
> > Any suggestions gratefully received.
> >
> > Mark Johnson
> >
>

Re: Message Group rebalancing on new consumers

[Justin Bertram <jb...@apache.org>]

What version of ActiveMQ are you using?


Justin

On Fri, Nov 25, 2022 at 4:20 AM Mark Johnson <sp...@gmail.com>
wrote:

> Hi,
>
> We have Apache Camel routes consuming messages and updating a database. Due
> to the nature of the business, there will be multiple messages in the queue
> that need to update the same row in the db tables. This results in a lot of
> db transaction rollbacks as the number of consumers increases.
>
> We have proved to ourselves that using Message Groups with part of the db
> table primary key as the JMSXGroupID resolves the rollback. There will also
> be considerably more groups than consumers, so I have no concerns about
> consumers being idle because all groups are already serviced.
>
> We have also proved that we can horizontally scale the Camel deployment
> based on queue depth.
>
> What we can't work out is how to put the two solutions together, as whilst
> Message Group will rebalance when a consumer disconnects, it does not
> rebalance on new consumers, according to the documentation.
>
> The documentation suggests that sending in a message with a -1 SeqNr will
> force the group to be treated as new, however at the point where we are
> scaling up there will already be thousands of messages on the queue.
>
> Does the reset happen when the broker receives the message or when the
> message gets to the head of the queue? If the latter, could I use message
> priority to force it to the head?
>
> If those conditions are met then I thinking that the Camel routes could
> process 1k msgs then send in a -1 reset message for their current GroupID
> to cause it to be rebalanced. Would also need to set prefetch=1 to ensure
> that the client does not have messages buffered, although even if we didn't
> do that, this pattern should reduce the number of rollbacks significantly.
>
> Is there a better way for groups to be moved to new consumers?
>
> Any suggestions gratefully received.
>
> Mark Johnson
>