Re: Message Groups and long running transactions

[George K <gk...@gmail.com>]

Hi,
If a solution has been identified on how to ensure that long running
transactions don't cause delays for messages from other message groups
please share.

Basically it would work if broker only assigned messages from a *new*
message group to a free consumer and never to a consumer that is busy
processing something already. So only messages form already assigned group
can be assigned prior to dispatch.

Without this working load balancing pretty much is broken for systems that
can encounter one off long running transactions.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-Groups-and-long-running-transactions-tp3499325p4691136.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Message Groups and long running transactions

[artnaseef <ar...@artnaseef.com>]

This is an interesting mix of needs.  Using message groups to ensure ordered
processing of messages while at least some of the messages take excessive
periods of time to process.

Can the slow-processing messages be separated from the message-groups?

Note the default message group handling uses a hash bucket of group
assignments to consumers, meaning multiple groups are assigned as a unit. 
So, assigning the slow messages to only one group will not prevent mixing
with fast messages (remember one consumer may be assigned multiple groups).

The -1 sequence on the group only takes effect as that message is dispatched
to the assigned owner (when it has prefetch space available), meaning it
goes to the same old owner as the message immediately before it, but then
new messages may go to a different owner.  So, actually, a higher prefetch
may be needed so that the -1 sequence message can be dispatched, but then it
will only affect messages that come after it, not those from before. 
Assigning a sequence number of -1 for the group on the slow messages would
do the trick - if combined with a prefect of 1.

Of course, if the fast and slow messages are in the same group, then the
fast messages need to wait for the slow ones to process.  That's the point
of message groups.

Another thought - is it possible eliminate the use of message groups adding
a message aggregator (camel) to collect all of the messages that require
sequential processing and feed them back out as a single message?




--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-Groups-and-long-running-transactions-tp3499325p4698769.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Message Groups and long running transactions

[George K <gk...@gmail.com>]

Thanks Hiram, I am pretty sure I tried that, but since then I went with a
custom approach so I can't verify it now. If I ever do, ill post back.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-Groups-and-long-running-transactions-tp3499325p4698128.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

Re: Message Groups and long running transactions

[Hiram Chirino <hi...@hiramchirino.com>]

Hi George,

If your consumer uses low prefetch (like 1), you should get the same effect.

On Thu, Feb 5, 2015 at 7:01 PM, George K <gk...@gmail.com> wrote:
> Hi,
> If a solution has been identified on how to ensure that long running
> transactions don't cause delays for messages from other message groups
> please share.
>
> Basically it would work if broker only assigned messages from a *new*
> message group to a free consumer and never to a consumer that is busy
> processing something already. So only messages form already assigned group
> can be assigned prior to dispatch.
>
> Without this working load balancing pretty much is broken for systems that
> can encounter one off long running transactions.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Message-Groups-and-long-running-transactions-tp3499325p4691136.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



-- 
Hiram Chirino
Engineering | Red Hat, Inc.
hchirino@redhat.com | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino