You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by Olivier Mallassi <ol...@gmail.com> on 2017/09/28 23:09:51 UTC
Consumer affinity (JMSXGroupID), AMQP 1.0
Gentleman,
I will need your help.
I have a use case where I would like to guarantee "consumer affinity",
which is usually implemented using the JMSXGroupID (actually, I am sure it
was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
My test does the "classical" case:
for (i ... i < 100.)
Message textMsg = new TextM....
textMsg.setStringProperty("JMSXGroupID", "groupA")
MessageProducer.send(textMsg);
and I start two consumers (assuming only one would work).
What I observe is that both consumers are receiving messages, acting as
competing consumers. I also tried added the qpid.group_header_key property
to the queue but it does not change anything.
My setup is
- java broker 6.0.4 & 6.1.4
- java clients using qpid-jms 0.25.0 (AMQP 1.0)
Is this the expected behaviour? Any ideas?
Complementary question: let's assume now, that there is the DR between the
broker & the consumers. Does the DR "propagate" or support "grouping
messages"? (I assume it is depending if you route the link or messages)
thanks for your help
Regards.
Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Posted by Rob Godfrey <ro...@gmail.com>.
On 30 September 2017 at 10:23, Olivier Mallassi <ol...@gmail.com>
wrote:
> Thanks all, it helps
> I saw this https://issues.apache.org/jira/browse/QPID-5165 while running
> the "test" but did not get the whole picture.
>
> Indeed, w/ multiple brokers + multiple DR, ensuring the consistency of the
> mapping group-id / consumers looks tricky (at least to me)...especially if
> you consider things like elasticity, network failures. no?
>
Yes - if you have multiple DRs then there would need to be some sort of
communication between them to share a common view of which groups go to
which brokers and this communication would need to survive failures / split
brain scenarios / etc.
Another potential avenue would be to somehow have the brokers allocate
groups to consumers in a deterministic manner; but this would only work if
the number (and identity) of the consumers was fixed and known beforehand.
-- Rob
>
> On Fri, Sep 29, 2017 at 4:10 PM, Adel Boutros <Ad...@live.com>
> wrote:
>
> > Thanks!
> >
> > ________________________________
> > From: Ken Giusti <kg...@redhat.com>
> > Sent: Friday, September 29, 2017 3:54:20 PM
> > To: users; keith.wall@gmail.com
> > Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
> >
> > At the moment the qpid dispatch router ignores all group related
> > message properties when computing the route for a destination.
> >
> > I've opened a corresponding feature request against the router to
> > track support for message groups:
> >
> > https://issues.apache.org/jira/browse/DISPATCH-843
> >
> > On Fri, Sep 29, 2017 at 9:23 AM, Keith W <ke...@gmail.com> wrote:
> > > A JIRA has been raised for this problem:
> > >
> > > https://issues.apache.org/jira/browse/QPID-7937
> > >
> > > On 29 September 2017 at 11:47, Rob Godfrey <ro...@gmail.com>
> > wrote:
> > >> On 29 September 2017 at 10:35, Adel Boutros <Ad...@live.com>
> > wrote:
> > >>
> > >>> Hello Rob,
> > >>>
> > >>>
> > >>> I would like to give my opinion on this.
> > >>>
> > >>>
> > >>> In our current use cases, we are configuring the brokers dynamically
> > using
> > >>> the REST API. We would like to have this possibility in the use case
> of
> > >>> Olivier.
> > >>
> > >>
> > >>> Also, having to restart the virtual host is damaging the High
> > Availability
> > >>> of the messaging system. This is because while the Virtual Host is
> > >>> restarting, no queues are available and the messages are
> inaccessible.
> > So I
> > >>> was wondering if restarting the virtual is a must or it could be
> fixed
> > in a
> > >>> Jira story?
> > >>>
> > >>>
> > >>
> > >> Sorry - my wording wasn't very clear. If you set this when you first
> > >> create the queue then you don't need to restart the vhost... however
> if
> > you
> > >> have an existing queue that you want to change, then the effect won't
> be
> > >> seen until the vhost is restarted (basically the queue sets itself up
> > to be
> > >> "group aware" or "not group aware" when it starts up, it can't change
> > while
> > >> it is running). I don't think this is really an issue for you in that
> > when
> > >> you create your queues with the REST API you just need to set this
> > >> attribute.
> > >>
> > >>
> > >>>
> > >>> Regarding the 2nd point of multiple brokers and dispatch router, if
> all
> > >>> brokers have the same queues configured with the appropriate grouping
> > >>> config, would that really break the feature?
> > >>>
> > >>> In our current use cases, all components have the same configuration
> > all
> > >>> the time (All brokers have same queues and same for the dispatch
> > router).
> > >>>
> > >>> So I would imagine if a consumer wants to consume a message with the
> > >>> correct group, the dispatch router will propagate this header to any
> > >>> available broker and will only get the corresponding messages.
> > >>>
> > >>>
> > >>>
> > >> The way grouping works is that the first time a message with a
> > particular
> > >> group id comes along, the broker will assign that group to a
> particular
> > >> consumer. Each broker will do this independently. So if you have two
> > >> brokers B1 and B2; and two consumers C1 and C2 and message M of
> group A
> > >> arrives on B1 whereas message N of group A arrives on B2; then B1 may
> > >> decide to associate group A with C1 whereas broker B2 decides to
> > associate
> > >> group A with consumer C2. So message groups with multiple brokers
> will
> > not
> > >> work behind a router *unless* the router is enhanced so that it is
> > aware of
> > >> message grouping (so all messages of the same group are sent to the
> same
> > >> broker) or the brokers somehow share information about the group ->
> > >> consumer assignment.
> > >>
> > >> -- Rob
> > >>
> > >> What do you think?
> > >>>
> > >>>
> > >>> Regards,
> > >>>
> > >>> Adel
> > >>>
> > >>> ________________________________
> > >>> From: Rob Godfrey <ro...@gmail.com>
> > >>> Sent: Friday, September 29, 2017 9:14:52 AM
> > >>> To: users@qpid.apache.org
> > >>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
> > >>>
> > >>> On 29 September 2017 at 01:09, Olivier Mallassi <
> > >>> olivier.mallassi@gmail.com>
> > >>> wrote:
> > >>>
> > >>> > Gentleman,
> > >>> >
> > >>> > I will need your help.
> > >>> >
> > >>> > I have a use case where I would like to guarantee "consumer
> > affinity",
> > >>> > which is usually implemented using the JMSXGroupID (actually, I am
> > sure
> > >>> it
> > >>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
> > >>> >
> > >>> > My test does the "classical" case:
> > >>> > for (i ... i < 100.)
> > >>> > Message textMsg = new TextM....
> > >>> > textMsg.setStringProperty("JMSXGroupID", "groupA")
> > >>> > MessageProducer.send(textMsg);
> > >>> >
> > >>> > and I start two consumers (assuming only one would work).
> > >>> >
> > >>> > What I observe is that both consumers are receiving messages,
> acting
> > as
> > >>> > competing consumers. I also tried added the qpid.group_header_key
> > >>> property
> > >>> > to the queue but it does not change anything.
> > >>> >
> > >>> >
> > >>> > My setup is
> > >>> > - java broker 6.0.4 & 6.1.4
> > >>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0)
> > >>> >
> > >>> > Is this the expected behaviour? Any ideas?
> > >>> >
> > >>> >
> > >>> This is a bug in the AMQP 1.0 behaviour of the broker. The Message
> > >>> Grouping functionality was originally built around the AMQP 0-x
> > protocols
> > >>> and is looking for a message group in the application headers of the
> > >>> message. In AMQP 1.0 there is a dedicated property for this and the
> > JMS
> > >>> client is (correctly from an AMQP 1.0 point of view) placing the
> group
> > id
> > >>> there.
> > >>>
> > >>> As a work around instead of using JMSXGroupID you could use any other
> > (non
> > >>> JMSX prefixed) header, e.g.:
> > >>>
> > >>> textMsg.setStringProperty("qpidBugWorkaround", "groupA")
> > >>>
> > >>> Note that on the broker you need to configure the queue so it knows
> > which
> > >>> header to use; this is stored in the messageGroupKey property of the
> > queue
> > >>> (to "qpidBugWorkaround" in this case) . Note that after setting this
> > >>> property you will need to restart the virtual host before the change
> > takes
> > >>> effect.
> > >>>
> > >>>
> > >>> >
> > >>> > Complementary question: let's assume now, that there is the DR
> > between
> > >>> the
> > >>> > broker & the consumers. Does the DR "propagate" or support
> "grouping
> > >>> > messages"? (I assume it is depending if you route the link or
> > messages)
> > >>> >
> > >>>
> > >>> With a single broker and link routing, yes grouping messages should
> > >>> continue to work.
> > >>>
> > >>> If there are multiple brokers sharding the queue then it would not
> > work -
> > >>> to support this the router would need to be changed to recognize
> > grouping
> > >>> and send all messages with the same group id to the same broker
> > waypoint
> > >>> (also, in this case, you would need to use message routing for
> inbound
> > >>> messages and link routing for consuming messages). We have talked
> > about
> > >>> support for behaviour like this before, but there is nothing
> > implemented
> > >>> yet as far as I know.
> > >>>
> > >>> -- Rob
> > >>>
> > >>>
> > >>> >
> > >>> >
> > >>> > thanks for your help
> > >>> >
> > >>> > Regards.
> > >>> >
> > >>>
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > > For additional commands, e-mail: users-help@qpid.apache.org
> > >
> >
> >
> >
> > --
> > -K
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
>
Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Posted by Robbie Gemmell <ro...@gmail.com>.
On 30 September 2017 at 09:23, Olivier Mallassi
<ol...@gmail.com> wrote:
> Thanks all, it helps
> I saw this https://issues.apache.org/jira/browse/QPID-5165 while running
> the "test" but did not get the whole picture.
>
The interesting bits here are all with the servers as already
discusse, but just to be clear...the above JIRA isn't related to the
current qpid-jms-client 0.25.0 release in any way, but rather a change
for an older prototype client that was included in the Qpid 0.26
release some years ago. The current AMQP 1.0 Qpid JMS client JIRAs are
at https://issues.apache.org/jira/browse/QPIDJMS
> Indeed, w/ multiple brokers + multiple DR, ensuring the consistency of the
> mapping group-id / consumers looks tricky (at least to me)...especially if
> you consider things like elasticity, network failures. no?
>
> On Fri, Sep 29, 2017 at 4:10 PM, Adel Boutros <Ad...@live.com> wrote:
>
>> Thanks!
>>
>> ________________________________
>> From: Ken Giusti <kg...@redhat.com>
>> Sent: Friday, September 29, 2017 3:54:20 PM
>> To: users; keith.wall@gmail.com
>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>>
>> At the moment the qpid dispatch router ignores all group related
>> message properties when computing the route for a destination.
>>
>> I've opened a corresponding feature request against the router to
>> track support for message groups:
>>
>> https://issues.apache.org/jira/browse/DISPATCH-843
>>
>> On Fri, Sep 29, 2017 at 9:23 AM, Keith W <ke...@gmail.com> wrote:
>> > A JIRA has been raised for this problem:
>> >
>> > https://issues.apache.org/jira/browse/QPID-7937
>> >
>> > On 29 September 2017 at 11:47, Rob Godfrey <ro...@gmail.com>
>> wrote:
>> >> On 29 September 2017 at 10:35, Adel Boutros <Ad...@live.com>
>> wrote:
>> >>
>> >>> Hello Rob,
>> >>>
>> >>>
>> >>> I would like to give my opinion on this.
>> >>>
>> >>>
>> >>> In our current use cases, we are configuring the brokers dynamically
>> using
>> >>> the REST API. We would like to have this possibility in the use case of
>> >>> Olivier.
>> >>
>> >>
>> >>> Also, having to restart the virtual host is damaging the High
>> Availability
>> >>> of the messaging system. This is because while the Virtual Host is
>> >>> restarting, no queues are available and the messages are inaccessible.
>> So I
>> >>> was wondering if restarting the virtual is a must or it could be fixed
>> in a
>> >>> Jira story?
>> >>>
>> >>>
>> >>
>> >> Sorry - my wording wasn't very clear. If you set this when you first
>> >> create the queue then you don't need to restart the vhost... however if
>> you
>> >> have an existing queue that you want to change, then the effect won't be
>> >> seen until the vhost is restarted (basically the queue sets itself up
>> to be
>> >> "group aware" or "not group aware" when it starts up, it can't change
>> while
>> >> it is running). I don't think this is really an issue for you in that
>> when
>> >> you create your queues with the REST API you just need to set this
>> >> attribute.
>> >>
>> >>
>> >>>
>> >>> Regarding the 2nd point of multiple brokers and dispatch router, if all
>> >>> brokers have the same queues configured with the appropriate grouping
>> >>> config, would that really break the feature?
>> >>>
>> >>> In our current use cases, all components have the same configuration
>> all
>> >>> the time (All brokers have same queues and same for the dispatch
>> router).
>> >>>
>> >>> So I would imagine if a consumer wants to consume a message with the
>> >>> correct group, the dispatch router will propagate this header to any
>> >>> available broker and will only get the corresponding messages.
>> >>>
>> >>>
>> >>>
>> >> The way grouping works is that the first time a message with a
>> particular
>> >> group id comes along, the broker will assign that group to a particular
>> >> consumer. Each broker will do this independently. So if you have two
>> >> brokers B1 and B2; and two consumers C1 and C2 and message M of group A
>> >> arrives on B1 whereas message N of group A arrives on B2; then B1 may
>> >> decide to associate group A with C1 whereas broker B2 decides to
>> associate
>> >> group A with consumer C2. So message groups with multiple brokers will
>> not
>> >> work behind a router *unless* the router is enhanced so that it is
>> aware of
>> >> message grouping (so all messages of the same group are sent to the same
>> >> broker) or the brokers somehow share information about the group ->
>> >> consumer assignment.
>> >>
>> >> -- Rob
>> >>
>> >> What do you think?
>> >>>
>> >>>
>> >>> Regards,
>> >>>
>> >>> Adel
>> >>>
>> >>> ________________________________
>> >>> From: Rob Godfrey <ro...@gmail.com>
>> >>> Sent: Friday, September 29, 2017 9:14:52 AM
>> >>> To: users@qpid.apache.org
>> >>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>> >>>
>> >>> On 29 September 2017 at 01:09, Olivier Mallassi <
>> >>> olivier.mallassi@gmail.com>
>> >>> wrote:
>> >>>
>> >>> > Gentleman,
>> >>> >
>> >>> > I will need your help.
>> >>> >
>> >>> > I have a use case where I would like to guarantee "consumer
>> affinity",
>> >>> > which is usually implemented using the JMSXGroupID (actually, I am
>> sure
>> >>> it
>> >>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
>> >>> >
>> >>> > My test does the "classical" case:
>> >>> > for (i ... i < 100.)
>> >>> > Message textMsg = new TextM....
>> >>> > textMsg.setStringProperty("JMSXGroupID", "groupA")
>> >>> > MessageProducer.send(textMsg);
>> >>> >
>> >>> > and I start two consumers (assuming only one would work).
>> >>> >
>> >>> > What I observe is that both consumers are receiving messages, acting
>> as
>> >>> > competing consumers. I also tried added the qpid.group_header_key
>> >>> property
>> >>> > to the queue but it does not change anything.
>> >>> >
>> >>> >
>> >>> > My setup is
>> >>> > - java broker 6.0.4 & 6.1.4
>> >>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0)
>> >>> >
>> >>> > Is this the expected behaviour? Any ideas?
>> >>> >
>> >>> >
>> >>> This is a bug in the AMQP 1.0 behaviour of the broker. The Message
>> >>> Grouping functionality was originally built around the AMQP 0-x
>> protocols
>> >>> and is looking for a message group in the application headers of the
>> >>> message. In AMQP 1.0 there is a dedicated property for this and the
>> JMS
>> >>> client is (correctly from an AMQP 1.0 point of view) placing the group
>> id
>> >>> there.
>> >>>
>> >>> As a work around instead of using JMSXGroupID you could use any other
>> (non
>> >>> JMSX prefixed) header, e.g.:
>> >>>
>> >>> textMsg.setStringProperty("qpidBugWorkaround", "groupA")
>> >>>
>> >>> Note that on the broker you need to configure the queue so it knows
>> which
>> >>> header to use; this is stored in the messageGroupKey property of the
>> queue
>> >>> (to "qpidBugWorkaround" in this case) . Note that after setting this
>> >>> property you will need to restart the virtual host before the change
>> takes
>> >>> effect.
>> >>>
>> >>>
>> >>> >
>> >>> > Complementary question: let's assume now, that there is the DR
>> between
>> >>> the
>> >>> > broker & the consumers. Does the DR "propagate" or support "grouping
>> >>> > messages"? (I assume it is depending if you route the link or
>> messages)
>> >>> >
>> >>>
>> >>> With a single broker and link routing, yes grouping messages should
>> >>> continue to work.
>> >>>
>> >>> If there are multiple brokers sharding the queue then it would not
>> work -
>> >>> to support this the router would need to be changed to recognize
>> grouping
>> >>> and send all messages with the same group id to the same broker
>> waypoint
>> >>> (also, in this case, you would need to use message routing for inbound
>> >>> messages and link routing for consuming messages). We have talked
>> about
>> >>> support for behaviour like this before, but there is nothing
>> implemented
>> >>> yet as far as I know.
>> >>>
>> >>> -- Rob
>> >>>
>> >>>
>> >>> >
>> >>> >
>> >>> > thanks for your help
>> >>> >
>> >>> > Regards.
>> >>> >
>> >>>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> > For additional commands, e-mail: users-help@qpid.apache.org
>> >
>>
>>
>>
>> --
>> -K
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>> For additional commands, e-mail: users-help@qpid.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Posted by Olivier Mallassi <ol...@gmail.com>.
Thanks all, it helps
I saw this https://issues.apache.org/jira/browse/QPID-5165 while running
the "test" but did not get the whole picture.
Indeed, w/ multiple brokers + multiple DR, ensuring the consistency of the
mapping group-id / consumers looks tricky (at least to me)...especially if
you consider things like elasticity, network failures. no?
On Fri, Sep 29, 2017 at 4:10 PM, Adel Boutros <Ad...@live.com> wrote:
> Thanks!
>
> ________________________________
> From: Ken Giusti <kg...@redhat.com>
> Sent: Friday, September 29, 2017 3:54:20 PM
> To: users; keith.wall@gmail.com
> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>
> At the moment the qpid dispatch router ignores all group related
> message properties when computing the route for a destination.
>
> I've opened a corresponding feature request against the router to
> track support for message groups:
>
> https://issues.apache.org/jira/browse/DISPATCH-843
>
> On Fri, Sep 29, 2017 at 9:23 AM, Keith W <ke...@gmail.com> wrote:
> > A JIRA has been raised for this problem:
> >
> > https://issues.apache.org/jira/browse/QPID-7937
> >
> > On 29 September 2017 at 11:47, Rob Godfrey <ro...@gmail.com>
> wrote:
> >> On 29 September 2017 at 10:35, Adel Boutros <Ad...@live.com>
> wrote:
> >>
> >>> Hello Rob,
> >>>
> >>>
> >>> I would like to give my opinion on this.
> >>>
> >>>
> >>> In our current use cases, we are configuring the brokers dynamically
> using
> >>> the REST API. We would like to have this possibility in the use case of
> >>> Olivier.
> >>
> >>
> >>> Also, having to restart the virtual host is damaging the High
> Availability
> >>> of the messaging system. This is because while the Virtual Host is
> >>> restarting, no queues are available and the messages are inaccessible.
> So I
> >>> was wondering if restarting the virtual is a must or it could be fixed
> in a
> >>> Jira story?
> >>>
> >>>
> >>
> >> Sorry - my wording wasn't very clear. If you set this when you first
> >> create the queue then you don't need to restart the vhost... however if
> you
> >> have an existing queue that you want to change, then the effect won't be
> >> seen until the vhost is restarted (basically the queue sets itself up
> to be
> >> "group aware" or "not group aware" when it starts up, it can't change
> while
> >> it is running). I don't think this is really an issue for you in that
> when
> >> you create your queues with the REST API you just need to set this
> >> attribute.
> >>
> >>
> >>>
> >>> Regarding the 2nd point of multiple brokers and dispatch router, if all
> >>> brokers have the same queues configured with the appropriate grouping
> >>> config, would that really break the feature?
> >>>
> >>> In our current use cases, all components have the same configuration
> all
> >>> the time (All brokers have same queues and same for the dispatch
> router).
> >>>
> >>> So I would imagine if a consumer wants to consume a message with the
> >>> correct group, the dispatch router will propagate this header to any
> >>> available broker and will only get the corresponding messages.
> >>>
> >>>
> >>>
> >> The way grouping works is that the first time a message with a
> particular
> >> group id comes along, the broker will assign that group to a particular
> >> consumer. Each broker will do this independently. So if you have two
> >> brokers B1 and B2; and two consumers C1 and C2 and message M of group A
> >> arrives on B1 whereas message N of group A arrives on B2; then B1 may
> >> decide to associate group A with C1 whereas broker B2 decides to
> associate
> >> group A with consumer C2. So message groups with multiple brokers will
> not
> >> work behind a router *unless* the router is enhanced so that it is
> aware of
> >> message grouping (so all messages of the same group are sent to the same
> >> broker) or the brokers somehow share information about the group ->
> >> consumer assignment.
> >>
> >> -- Rob
> >>
> >> What do you think?
> >>>
> >>>
> >>> Regards,
> >>>
> >>> Adel
> >>>
> >>> ________________________________
> >>> From: Rob Godfrey <ro...@gmail.com>
> >>> Sent: Friday, September 29, 2017 9:14:52 AM
> >>> To: users@qpid.apache.org
> >>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
> >>>
> >>> On 29 September 2017 at 01:09, Olivier Mallassi <
> >>> olivier.mallassi@gmail.com>
> >>> wrote:
> >>>
> >>> > Gentleman,
> >>> >
> >>> > I will need your help.
> >>> >
> >>> > I have a use case where I would like to guarantee "consumer
> affinity",
> >>> > which is usually implemented using the JMSXGroupID (actually, I am
> sure
> >>> it
> >>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
> >>> >
> >>> > My test does the "classical" case:
> >>> > for (i ... i < 100.)
> >>> > Message textMsg = new TextM....
> >>> > textMsg.setStringProperty("JMSXGroupID", "groupA")
> >>> > MessageProducer.send(textMsg);
> >>> >
> >>> > and I start two consumers (assuming only one would work).
> >>> >
> >>> > What I observe is that both consumers are receiving messages, acting
> as
> >>> > competing consumers. I also tried added the qpid.group_header_key
> >>> property
> >>> > to the queue but it does not change anything.
> >>> >
> >>> >
> >>> > My setup is
> >>> > - java broker 6.0.4 & 6.1.4
> >>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0)
> >>> >
> >>> > Is this the expected behaviour? Any ideas?
> >>> >
> >>> >
> >>> This is a bug in the AMQP 1.0 behaviour of the broker. The Message
> >>> Grouping functionality was originally built around the AMQP 0-x
> protocols
> >>> and is looking for a message group in the application headers of the
> >>> message. In AMQP 1.0 there is a dedicated property for this and the
> JMS
> >>> client is (correctly from an AMQP 1.0 point of view) placing the group
> id
> >>> there.
> >>>
> >>> As a work around instead of using JMSXGroupID you could use any other
> (non
> >>> JMSX prefixed) header, e.g.:
> >>>
> >>> textMsg.setStringProperty("qpidBugWorkaround", "groupA")
> >>>
> >>> Note that on the broker you need to configure the queue so it knows
> which
> >>> header to use; this is stored in the messageGroupKey property of the
> queue
> >>> (to "qpidBugWorkaround" in this case) . Note that after setting this
> >>> property you will need to restart the virtual host before the change
> takes
> >>> effect.
> >>>
> >>>
> >>> >
> >>> > Complementary question: let's assume now, that there is the DR
> between
> >>> the
> >>> > broker & the consumers. Does the DR "propagate" or support "grouping
> >>> > messages"? (I assume it is depending if you route the link or
> messages)
> >>> >
> >>>
> >>> With a single broker and link routing, yes grouping messages should
> >>> continue to work.
> >>>
> >>> If there are multiple brokers sharding the queue then it would not
> work -
> >>> to support this the router would need to be changed to recognize
> grouping
> >>> and send all messages with the same group id to the same broker
> waypoint
> >>> (also, in this case, you would need to use message routing for inbound
> >>> messages and link routing for consuming messages). We have talked
> about
> >>> support for behaviour like this before, but there is nothing
> implemented
> >>> yet as far as I know.
> >>>
> >>> -- Rob
> >>>
> >>>
> >>> >
> >>> >
> >>> > thanks for your help
> >>> >
> >>> > Regards.
> >>> >
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
>
>
>
> --
> -K
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Posted by Adel Boutros <Ad...@live.com>.
Thanks!
________________________________
From: Ken Giusti <kg...@redhat.com>
Sent: Friday, September 29, 2017 3:54:20 PM
To: users; keith.wall@gmail.com
Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
At the moment the qpid dispatch router ignores all group related
message properties when computing the route for a destination.
I've opened a corresponding feature request against the router to
track support for message groups:
https://issues.apache.org/jira/browse/DISPATCH-843
On Fri, Sep 29, 2017 at 9:23 AM, Keith W <ke...@gmail.com> wrote:
> A JIRA has been raised for this problem:
>
> https://issues.apache.org/jira/browse/QPID-7937
>
> On 29 September 2017 at 11:47, Rob Godfrey <ro...@gmail.com> wrote:
>> On 29 September 2017 at 10:35, Adel Boutros <Ad...@live.com> wrote:
>>
>>> Hello Rob,
>>>
>>>
>>> I would like to give my opinion on this.
>>>
>>>
>>> In our current use cases, we are configuring the brokers dynamically using
>>> the REST API. We would like to have this possibility in the use case of
>>> Olivier.
>>
>>
>>> Also, having to restart the virtual host is damaging the High Availability
>>> of the messaging system. This is because while the Virtual Host is
>>> restarting, no queues are available and the messages are inaccessible. So I
>>> was wondering if restarting the virtual is a must or it could be fixed in a
>>> Jira story?
>>>
>>>
>>
>> Sorry - my wording wasn't very clear. If you set this when you first
>> create the queue then you don't need to restart the vhost... however if you
>> have an existing queue that you want to change, then the effect won't be
>> seen until the vhost is restarted (basically the queue sets itself up to be
>> "group aware" or "not group aware" when it starts up, it can't change while
>> it is running). I don't think this is really an issue for you in that when
>> you create your queues with the REST API you just need to set this
>> attribute.
>>
>>
>>>
>>> Regarding the 2nd point of multiple brokers and dispatch router, if all
>>> brokers have the same queues configured with the appropriate grouping
>>> config, would that really break the feature?
>>>
>>> In our current use cases, all components have the same configuration all
>>> the time (All brokers have same queues and same for the dispatch router).
>>>
>>> So I would imagine if a consumer wants to consume a message with the
>>> correct group, the dispatch router will propagate this header to any
>>> available broker and will only get the corresponding messages.
>>>
>>>
>>>
>> The way grouping works is that the first time a message with a particular
>> group id comes along, the broker will assign that group to a particular
>> consumer. Each broker will do this independently. So if you have two
>> brokers B1 and B2; and two consumers C1 and C2 and message M of group A
>> arrives on B1 whereas message N of group A arrives on B2; then B1 may
>> decide to associate group A with C1 whereas broker B2 decides to associate
>> group A with consumer C2. So message groups with multiple brokers will not
>> work behind a router *unless* the router is enhanced so that it is aware of
>> message grouping (so all messages of the same group are sent to the same
>> broker) or the brokers somehow share information about the group ->
>> consumer assignment.
>>
>> -- Rob
>>
>> What do you think?
>>>
>>>
>>> Regards,
>>>
>>> Adel
>>>
>>> ________________________________
>>> From: Rob Godfrey <ro...@gmail.com>
>>> Sent: Friday, September 29, 2017 9:14:52 AM
>>> To: users@qpid.apache.org
>>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>>>
>>> On 29 September 2017 at 01:09, Olivier Mallassi <
>>> olivier.mallassi@gmail.com>
>>> wrote:
>>>
>>> > Gentleman,
>>> >
>>> > I will need your help.
>>> >
>>> > I have a use case where I would like to guarantee "consumer affinity",
>>> > which is usually implemented using the JMSXGroupID (actually, I am sure
>>> it
>>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
>>> >
>>> > My test does the "classical" case:
>>> > for (i ... i < 100.)
>>> > Message textMsg = new TextM....
>>> > textMsg.setStringProperty("JMSXGroupID", "groupA")
>>> > MessageProducer.send(textMsg);
>>> >
>>> > and I start two consumers (assuming only one would work).
>>> >
>>> > What I observe is that both consumers are receiving messages, acting as
>>> > competing consumers. I also tried added the qpid.group_header_key
>>> property
>>> > to the queue but it does not change anything.
>>> >
>>> >
>>> > My setup is
>>> > - java broker 6.0.4 & 6.1.4
>>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0)
>>> >
>>> > Is this the expected behaviour? Any ideas?
>>> >
>>> >
>>> This is a bug in the AMQP 1.0 behaviour of the broker. The Message
>>> Grouping functionality was originally built around the AMQP 0-x protocols
>>> and is looking for a message group in the application headers of the
>>> message. In AMQP 1.0 there is a dedicated property for this and the JMS
>>> client is (correctly from an AMQP 1.0 point of view) placing the group id
>>> there.
>>>
>>> As a work around instead of using JMSXGroupID you could use any other (non
>>> JMSX prefixed) header, e.g.:
>>>
>>> textMsg.setStringProperty("qpidBugWorkaround", "groupA")
>>>
>>> Note that on the broker you need to configure the queue so it knows which
>>> header to use; this is stored in the messageGroupKey property of the queue
>>> (to "qpidBugWorkaround" in this case) . Note that after setting this
>>> property you will need to restart the virtual host before the change takes
>>> effect.
>>>
>>>
>>> >
>>> > Complementary question: let's assume now, that there is the DR between
>>> the
>>> > broker & the consumers. Does the DR "propagate" or support "grouping
>>> > messages"? (I assume it is depending if you route the link or messages)
>>> >
>>>
>>> With a single broker and link routing, yes grouping messages should
>>> continue to work.
>>>
>>> If there are multiple brokers sharding the queue then it would not work -
>>> to support this the router would need to be changed to recognize grouping
>>> and send all messages with the same group id to the same broker waypoint
>>> (also, in this case, you would need to use message routing for inbound
>>> messages and link routing for consuming messages). We have talked about
>>> support for behaviour like this before, but there is nothing implemented
>>> yet as far as I know.
>>>
>>> -- Rob
>>>
>>>
>>> >
>>> >
>>> > thanks for your help
>>> >
>>> > Regards.
>>> >
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Posted by Ken Giusti <kg...@redhat.com>.
At the moment the qpid dispatch router ignores all group related
message properties when computing the route for a destination.
I've opened a corresponding feature request against the router to
track support for message groups:
https://issues.apache.org/jira/browse/DISPATCH-843
On Fri, Sep 29, 2017 at 9:23 AM, Keith W <ke...@gmail.com> wrote:
> A JIRA has been raised for this problem:
>
> https://issues.apache.org/jira/browse/QPID-7937
>
> On 29 September 2017 at 11:47, Rob Godfrey <ro...@gmail.com> wrote:
>> On 29 September 2017 at 10:35, Adel Boutros <Ad...@live.com> wrote:
>>
>>> Hello Rob,
>>>
>>>
>>> I would like to give my opinion on this.
>>>
>>>
>>> In our current use cases, we are configuring the brokers dynamically using
>>> the REST API. We would like to have this possibility in the use case of
>>> Olivier.
>>
>>
>>> Also, having to restart the virtual host is damaging the High Availability
>>> of the messaging system. This is because while the Virtual Host is
>>> restarting, no queues are available and the messages are inaccessible. So I
>>> was wondering if restarting the virtual is a must or it could be fixed in a
>>> Jira story?
>>>
>>>
>>
>> Sorry - my wording wasn't very clear. If you set this when you first
>> create the queue then you don't need to restart the vhost... however if you
>> have an existing queue that you want to change, then the effect won't be
>> seen until the vhost is restarted (basically the queue sets itself up to be
>> "group aware" or "not group aware" when it starts up, it can't change while
>> it is running). I don't think this is really an issue for you in that when
>> you create your queues with the REST API you just need to set this
>> attribute.
>>
>>
>>>
>>> Regarding the 2nd point of multiple brokers and dispatch router, if all
>>> brokers have the same queues configured with the appropriate grouping
>>> config, would that really break the feature?
>>>
>>> In our current use cases, all components have the same configuration all
>>> the time (All brokers have same queues and same for the dispatch router).
>>>
>>> So I would imagine if a consumer wants to consume a message with the
>>> correct group, the dispatch router will propagate this header to any
>>> available broker and will only get the corresponding messages.
>>>
>>>
>>>
>> The way grouping works is that the first time a message with a particular
>> group id comes along, the broker will assign that group to a particular
>> consumer. Each broker will do this independently. So if you have two
>> brokers B1 and B2; and two consumers C1 and C2 and message M of group A
>> arrives on B1 whereas message N of group A arrives on B2; then B1 may
>> decide to associate group A with C1 whereas broker B2 decides to associate
>> group A with consumer C2. So message groups with multiple brokers will not
>> work behind a router *unless* the router is enhanced so that it is aware of
>> message grouping (so all messages of the same group are sent to the same
>> broker) or the brokers somehow share information about the group ->
>> consumer assignment.
>>
>> -- Rob
>>
>> What do you think?
>>>
>>>
>>> Regards,
>>>
>>> Adel
>>>
>>> ________________________________
>>> From: Rob Godfrey <ro...@gmail.com>
>>> Sent: Friday, September 29, 2017 9:14:52 AM
>>> To: users@qpid.apache.org
>>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>>>
>>> On 29 September 2017 at 01:09, Olivier Mallassi <
>>> olivier.mallassi@gmail.com>
>>> wrote:
>>>
>>> > Gentleman,
>>> >
>>> > I will need your help.
>>> >
>>> > I have a use case where I would like to guarantee "consumer affinity",
>>> > which is usually implemented using the JMSXGroupID (actually, I am sure
>>> it
>>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
>>> >
>>> > My test does the "classical" case:
>>> > for (i ... i < 100.)
>>> > Message textMsg = new TextM....
>>> > textMsg.setStringProperty("JMSXGroupID", "groupA")
>>> > MessageProducer.send(textMsg);
>>> >
>>> > and I start two consumers (assuming only one would work).
>>> >
>>> > What I observe is that both consumers are receiving messages, acting as
>>> > competing consumers. I also tried added the qpid.group_header_key
>>> property
>>> > to the queue but it does not change anything.
>>> >
>>> >
>>> > My setup is
>>> > - java broker 6.0.4 & 6.1.4
>>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0)
>>> >
>>> > Is this the expected behaviour? Any ideas?
>>> >
>>> >
>>> This is a bug in the AMQP 1.0 behaviour of the broker. The Message
>>> Grouping functionality was originally built around the AMQP 0-x protocols
>>> and is looking for a message group in the application headers of the
>>> message. In AMQP 1.0 there is a dedicated property for this and the JMS
>>> client is (correctly from an AMQP 1.0 point of view) placing the group id
>>> there.
>>>
>>> As a work around instead of using JMSXGroupID you could use any other (non
>>> JMSX prefixed) header, e.g.:
>>>
>>> textMsg.setStringProperty("qpidBugWorkaround", "groupA")
>>>
>>> Note that on the broker you need to configure the queue so it knows which
>>> header to use; this is stored in the messageGroupKey property of the queue
>>> (to "qpidBugWorkaround" in this case) . Note that after setting this
>>> property you will need to restart the virtual host before the change takes
>>> effect.
>>>
>>>
>>> >
>>> > Complementary question: let's assume now, that there is the DR between
>>> the
>>> > broker & the consumers. Does the DR "propagate" or support "grouping
>>> > messages"? (I assume it is depending if you route the link or messages)
>>> >
>>>
>>> With a single broker and link routing, yes grouping messages should
>>> continue to work.
>>>
>>> If there are multiple brokers sharding the queue then it would not work -
>>> to support this the router would need to be changed to recognize grouping
>>> and send all messages with the same group id to the same broker waypoint
>>> (also, in this case, you would need to use message routing for inbound
>>> messages and link routing for consuming messages). We have talked about
>>> support for behaviour like this before, but there is nothing implemented
>>> yet as far as I know.
>>>
>>> -- Rob
>>>
>>>
>>> >
>>> >
>>> > thanks for your help
>>> >
>>> > Regards.
>>> >
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
--
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Posted by Keith W <ke...@gmail.com>.
A JIRA has been raised for this problem:
https://issues.apache.org/jira/browse/QPID-7937
On 29 September 2017 at 11:47, Rob Godfrey <ro...@gmail.com> wrote:
> On 29 September 2017 at 10:35, Adel Boutros <Ad...@live.com> wrote:
>
>> Hello Rob,
>>
>>
>> I would like to give my opinion on this.
>>
>>
>> In our current use cases, we are configuring the brokers dynamically using
>> the REST API. We would like to have this possibility in the use case of
>> Olivier.
>
>
>> Also, having to restart the virtual host is damaging the High Availability
>> of the messaging system. This is because while the Virtual Host is
>> restarting, no queues are available and the messages are inaccessible. So I
>> was wondering if restarting the virtual is a must or it could be fixed in a
>> Jira story?
>>
>>
>
> Sorry - my wording wasn't very clear. If you set this when you first
> create the queue then you don't need to restart the vhost... however if you
> have an existing queue that you want to change, then the effect won't be
> seen until the vhost is restarted (basically the queue sets itself up to be
> "group aware" or "not group aware" when it starts up, it can't change while
> it is running). I don't think this is really an issue for you in that when
> you create your queues with the REST API you just need to set this
> attribute.
>
>
>>
>> Regarding the 2nd point of multiple brokers and dispatch router, if all
>> brokers have the same queues configured with the appropriate grouping
>> config, would that really break the feature?
>>
>> In our current use cases, all components have the same configuration all
>> the time (All brokers have same queues and same for the dispatch router).
>>
>> So I would imagine if a consumer wants to consume a message with the
>> correct group, the dispatch router will propagate this header to any
>> available broker and will only get the corresponding messages.
>>
>>
>>
> The way grouping works is that the first time a message with a particular
> group id comes along, the broker will assign that group to a particular
> consumer. Each broker will do this independently. So if you have two
> brokers B1 and B2; and two consumers C1 and C2 and message M of group A
> arrives on B1 whereas message N of group A arrives on B2; then B1 may
> decide to associate group A with C1 whereas broker B2 decides to associate
> group A with consumer C2. So message groups with multiple brokers will not
> work behind a router *unless* the router is enhanced so that it is aware of
> message grouping (so all messages of the same group are sent to the same
> broker) or the brokers somehow share information about the group ->
> consumer assignment.
>
> -- Rob
>
> What do you think?
>>
>>
>> Regards,
>>
>> Adel
>>
>> ________________________________
>> From: Rob Godfrey <ro...@gmail.com>
>> Sent: Friday, September 29, 2017 9:14:52 AM
>> To: users@qpid.apache.org
>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>>
>> On 29 September 2017 at 01:09, Olivier Mallassi <
>> olivier.mallassi@gmail.com>
>> wrote:
>>
>> > Gentleman,
>> >
>> > I will need your help.
>> >
>> > I have a use case where I would like to guarantee "consumer affinity",
>> > which is usually implemented using the JMSXGroupID (actually, I am sure
>> it
>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
>> >
>> > My test does the "classical" case:
>> > for (i ... i < 100.)
>> > Message textMsg = new TextM....
>> > textMsg.setStringProperty("JMSXGroupID", "groupA")
>> > MessageProducer.send(textMsg);
>> >
>> > and I start two consumers (assuming only one would work).
>> >
>> > What I observe is that both consumers are receiving messages, acting as
>> > competing consumers. I also tried added the qpid.group_header_key
>> property
>> > to the queue but it does not change anything.
>> >
>> >
>> > My setup is
>> > - java broker 6.0.4 & 6.1.4
>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0)
>> >
>> > Is this the expected behaviour? Any ideas?
>> >
>> >
>> This is a bug in the AMQP 1.0 behaviour of the broker. The Message
>> Grouping functionality was originally built around the AMQP 0-x protocols
>> and is looking for a message group in the application headers of the
>> message. In AMQP 1.0 there is a dedicated property for this and the JMS
>> client is (correctly from an AMQP 1.0 point of view) placing the group id
>> there.
>>
>> As a work around instead of using JMSXGroupID you could use any other (non
>> JMSX prefixed) header, e.g.:
>>
>> textMsg.setStringProperty("qpidBugWorkaround", "groupA")
>>
>> Note that on the broker you need to configure the queue so it knows which
>> header to use; this is stored in the messageGroupKey property of the queue
>> (to "qpidBugWorkaround" in this case) . Note that after setting this
>> property you will need to restart the virtual host before the change takes
>> effect.
>>
>>
>> >
>> > Complementary question: let's assume now, that there is the DR between
>> the
>> > broker & the consumers. Does the DR "propagate" or support "grouping
>> > messages"? (I assume it is depending if you route the link or messages)
>> >
>>
>> With a single broker and link routing, yes grouping messages should
>> continue to work.
>>
>> If there are multiple brokers sharding the queue then it would not work -
>> to support this the router would need to be changed to recognize grouping
>> and send all messages with the same group id to the same broker waypoint
>> (also, in this case, you would need to use message routing for inbound
>> messages and link routing for consuming messages). We have talked about
>> support for behaviour like this before, but there is nothing implemented
>> yet as far as I know.
>>
>> -- Rob
>>
>>
>> >
>> >
>> > thanks for your help
>> >
>> > Regards.
>> >
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Posted by Rob Godfrey <ro...@gmail.com>.
On 29 September 2017 at 10:35, Adel Boutros <Ad...@live.com> wrote:
> Hello Rob,
>
>
> I would like to give my opinion on this.
>
>
> In our current use cases, we are configuring the brokers dynamically using
> the REST API. We would like to have this possibility in the use case of
> Olivier.
> Also, having to restart the virtual host is damaging the High Availability
> of the messaging system. This is because while the Virtual Host is
> restarting, no queues are available and the messages are inaccessible. So I
> was wondering if restarting the virtual is a must or it could be fixed in a
> Jira story?
>
>
Sorry - my wording wasn't very clear. If you set this when you first
create the queue then you don't need to restart the vhost... however if you
have an existing queue that you want to change, then the effect won't be
seen until the vhost is restarted (basically the queue sets itself up to be
"group aware" or "not group aware" when it starts up, it can't change while
it is running). I don't think this is really an issue for you in that when
you create your queues with the REST API you just need to set this
attribute.
>
> Regarding the 2nd point of multiple brokers and dispatch router, if all
> brokers have the same queues configured with the appropriate grouping
> config, would that really break the feature?
>
> In our current use cases, all components have the same configuration all
> the time (All brokers have same queues and same for the dispatch router).
>
> So I would imagine if a consumer wants to consume a message with the
> correct group, the dispatch router will propagate this header to any
> available broker and will only get the corresponding messages.
>
>
>
The way grouping works is that the first time a message with a particular
group id comes along, the broker will assign that group to a particular
consumer. Each broker will do this independently. So if you have two
brokers B1 and B2; and two consumers C1 and C2 and message M of group A
arrives on B1 whereas message N of group A arrives on B2; then B1 may
decide to associate group A with C1 whereas broker B2 decides to associate
group A with consumer C2. So message groups with multiple brokers will not
work behind a router *unless* the router is enhanced so that it is aware of
message grouping (so all messages of the same group are sent to the same
broker) or the brokers somehow share information about the group ->
consumer assignment.
-- Rob
What do you think?
>
>
> Regards,
>
> Adel
>
> ________________________________
> From: Rob Godfrey <ro...@gmail.com>
> Sent: Friday, September 29, 2017 9:14:52 AM
> To: users@qpid.apache.org
> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
>
> On 29 September 2017 at 01:09, Olivier Mallassi <
> olivier.mallassi@gmail.com>
> wrote:
>
> > Gentleman,
> >
> > I will need your help.
> >
> > I have a use case where I would like to guarantee "consumer affinity",
> > which is usually implemented using the JMSXGroupID (actually, I am sure
> it
> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
> >
> > My test does the "classical" case:
> > for (i ... i < 100.)
> > Message textMsg = new TextM....
> > textMsg.setStringProperty("JMSXGroupID", "groupA")
> > MessageProducer.send(textMsg);
> >
> > and I start two consumers (assuming only one would work).
> >
> > What I observe is that both consumers are receiving messages, acting as
> > competing consumers. I also tried added the qpid.group_header_key
> property
> > to the queue but it does not change anything.
> >
> >
> > My setup is
> > - java broker 6.0.4 & 6.1.4
> > - java clients using qpid-jms 0.25.0 (AMQP 1.0)
> >
> > Is this the expected behaviour? Any ideas?
> >
> >
> This is a bug in the AMQP 1.0 behaviour of the broker. The Message
> Grouping functionality was originally built around the AMQP 0-x protocols
> and is looking for a message group in the application headers of the
> message. In AMQP 1.0 there is a dedicated property for this and the JMS
> client is (correctly from an AMQP 1.0 point of view) placing the group id
> there.
>
> As a work around instead of using JMSXGroupID you could use any other (non
> JMSX prefixed) header, e.g.:
>
> textMsg.setStringProperty("qpidBugWorkaround", "groupA")
>
> Note that on the broker you need to configure the queue so it knows which
> header to use; this is stored in the messageGroupKey property of the queue
> (to "qpidBugWorkaround" in this case) . Note that after setting this
> property you will need to restart the virtual host before the change takes
> effect.
>
>
> >
> > Complementary question: let's assume now, that there is the DR between
> the
> > broker & the consumers. Does the DR "propagate" or support "grouping
> > messages"? (I assume it is depending if you route the link or messages)
> >
>
> With a single broker and link routing, yes grouping messages should
> continue to work.
>
> If there are multiple brokers sharding the queue then it would not work -
> to support this the router would need to be changed to recognize grouping
> and send all messages with the same group id to the same broker waypoint
> (also, in this case, you would need to use message routing for inbound
> messages and link routing for consuming messages). We have talked about
> support for behaviour like this before, but there is nothing implemented
> yet as far as I know.
>
> -- Rob
>
>
> >
> >
> > thanks for your help
> >
> > Regards.
> >
>
Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Posted by Adel Boutros <Ad...@live.com>.
Hello Rob,
I would like to give my opinion on this.
In our current use cases, we are configuring the brokers dynamically using the REST API. We would like to have this possibility in the use case of Olivier.
Also, having to restart the virtual host is damaging the High Availability of the messaging system. This is because while the Virtual Host is restarting, no queues are available and the messages are inaccessible. So I was wondering if restarting the virtual is a must or it could be fixed in a Jira story?
Regarding the 2nd point of multiple brokers and dispatch router, if all brokers have the same queues configured with the appropriate grouping config, would that really break the feature?
In our current use cases, all components have the same configuration all the time (All brokers have same queues and same for the dispatch router).
So I would imagine if a consumer wants to consume a message with the correct group, the dispatch router will propagate this header to any available broker and will only get the corresponding messages.
What do you think?
Regards,
Adel
________________________________
From: Rob Godfrey <ro...@gmail.com>
Sent: Friday, September 29, 2017 9:14:52 AM
To: users@qpid.apache.org
Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0
On 29 September 2017 at 01:09, Olivier Mallassi <ol...@gmail.com>
wrote:
> Gentleman,
>
> I will need your help.
>
> I have a use case where I would like to guarantee "consumer affinity",
> which is usually implemented using the JMSXGroupID (actually, I am sure it
> was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
>
> My test does the "classical" case:
> for (i ... i < 100.)
> Message textMsg = new TextM....
> textMsg.setStringProperty("JMSXGroupID", "groupA")
> MessageProducer.send(textMsg);
>
> and I start two consumers (assuming only one would work).
>
> What I observe is that both consumers are receiving messages, acting as
> competing consumers. I also tried added the qpid.group_header_key property
> to the queue but it does not change anything.
>
>
> My setup is
> - java broker 6.0.4 & 6.1.4
> - java clients using qpid-jms 0.25.0 (AMQP 1.0)
>
> Is this the expected behaviour? Any ideas?
>
>
This is a bug in the AMQP 1.0 behaviour of the broker. The Message
Grouping functionality was originally built around the AMQP 0-x protocols
and is looking for a message group in the application headers of the
message. In AMQP 1.0 there is a dedicated property for this and the JMS
client is (correctly from an AMQP 1.0 point of view) placing the group id
there.
As a work around instead of using JMSXGroupID you could use any other (non
JMSX prefixed) header, e.g.:
textMsg.setStringProperty("qpidBugWorkaround", "groupA")
Note that on the broker you need to configure the queue so it knows which
header to use; this is stored in the messageGroupKey property of the queue
(to "qpidBugWorkaround" in this case) . Note that after setting this
property you will need to restart the virtual host before the change takes
effect.
>
> Complementary question: let's assume now, that there is the DR between the
> broker & the consumers. Does the DR "propagate" or support "grouping
> messages"? (I assume it is depending if you route the link or messages)
>
With a single broker and link routing, yes grouping messages should
continue to work.
If there are multiple brokers sharding the queue then it would not work -
to support this the router would need to be changed to recognize grouping
and send all messages with the same group id to the same broker waypoint
(also, in this case, you would need to use message routing for inbound
messages and link routing for consuming messages). We have talked about
support for behaviour like this before, but there is nothing implemented
yet as far as I know.
-- Rob
>
>
> thanks for your help
>
> Regards.
>
Re: Consumer affinity (JMSXGroupID), AMQP 1.0
Posted by Rob Godfrey <ro...@gmail.com>.
On 29 September 2017 at 01:09, Olivier Mallassi <ol...@gmail.com>
wrote:
> Gentleman,
>
> I will need your help.
>
> I have a use case where I would like to guarantee "consumer affinity",
> which is usually implemented using the JMSXGroupID (actually, I am sure it
> was working w/ AMQP 0.10 clients but here I am using AMQP 1.0)
>
> My test does the "classical" case:
> for (i ... i < 100.)
> Message textMsg = new TextM....
> textMsg.setStringProperty("JMSXGroupID", "groupA")
> MessageProducer.send(textMsg);
>
> and I start two consumers (assuming only one would work).
>
> What I observe is that both consumers are receiving messages, acting as
> competing consumers. I also tried added the qpid.group_header_key property
> to the queue but it does not change anything.
>
>
> My setup is
> - java broker 6.0.4 & 6.1.4
> - java clients using qpid-jms 0.25.0 (AMQP 1.0)
>
> Is this the expected behaviour? Any ideas?
>
>
This is a bug in the AMQP 1.0 behaviour of the broker. The Message
Grouping functionality was originally built around the AMQP 0-x protocols
and is looking for a message group in the application headers of the
message. In AMQP 1.0 there is a dedicated property for this and the JMS
client is (correctly from an AMQP 1.0 point of view) placing the group id
there.
As a work around instead of using JMSXGroupID you could use any other (non
JMSX prefixed) header, e.g.:
textMsg.setStringProperty("qpidBugWorkaround", "groupA")
Note that on the broker you need to configure the queue so it knows which
header to use; this is stored in the messageGroupKey property of the queue
(to "qpidBugWorkaround" in this case) . Note that after setting this
property you will need to restart the virtual host before the change takes
effect.
>
> Complementary question: let's assume now, that there is the DR between the
> broker & the consumers. Does the DR "propagate" or support "grouping
> messages"? (I assume it is depending if you route the link or messages)
>
With a single broker and link routing, yes grouping messages should
continue to work.
If there are multiple brokers sharding the queue then it would not work -
to support this the router would need to be changed to recognize grouping
and send all messages with the same group id to the same broker waypoint
(also, in this case, you would need to use message routing for inbound
messages and link routing for consuming messages). We have talked about
support for behaviour like this before, but there is nothing implemented
yet as far as I know.
-- Rob
>
>
> thanks for your help
>
> Regards.
>