Re: 1 Queue with 2 Consumers - turn off pre-fetching?

[Helen Kwong <he...@gmail.com>]

Hi Rob,

We have implemented the workaround Dan mentioned before -- by suspending
the session with sendSuspendChannel() and releasing prefetched messages
with rejectMessagesForConsumerTag() whenever we detect a long-running
message -- and so far it is working as expected. Thank you again for all
your help.

For the long term solution, we plan to upgrade our client and broker to the
latest version, and use the "single consumer across many queues" feature
that you're building. So I'm wondering:
1. Do you still plan to have this available in v0.32 of Qpid?
2. What is the planned release date of v0.32?

Thanks!

Helen

On Tue, Sep 9, 2014 at 12:54 PM, xiaodan.wang <xi...@salesforce.com>
wrote:

> Hi Rob, in case you are interested, wanted to give you an update from our
> side. First, let me begin by saying how awesome the Qpid community has
> been.
> Your patience and responsiveness in addressing our questions is way beyond
> what we could have hoped for from this community. I think I speak for
> Salesforce in saying that we owe you a couple rounds of beer, in case you
> are ever bored and in San Francisco :)
>
> Our plan is to address per-consumer prefetching on the v0.16 clients we
> currently run in production using the suspend & release prefetched messages
> approach described above. The general flow of our solution is as follows:
>
> 1) Each time a message is dispatched and onMessage is invoked, we register
> the parent Qpid session with a session tracker
>
> 2)  The session tracker is scheduled every x seconds and looks for Qpid
> sessions that is processing a long running messages
>
> 3) Once the session tracker finds a long running message, it invokes
> "sendSuspendChannel" and "rejectMessagesForConsumerTag" on the session to
> suspend dequeue and release any prefetched messages
>
> 4) When the long running message is completed, it unregisters the parent
> Qpid session from the session tracker and invokes "sendSuspendChannel" to
> resume dequeue
>
> We are in the process of performance testing this solution to make sure
> that
> a) overhead of suspend and release is manageable and b) no anomalies arise
> from repeatedly suspending and unsuspending the session
>
>
> For our long term plan, we plan to upgrade to the latest version of the
> client/broker and use the single-consumer-multiple-queue feature you've
> built in v0.32 for AMQP 0-10. After we've patched our v0.16 client, we plan
> to focus on vetting this.
>
> We are also abandoning efforts to switch to per-session prefetch with AMQP
> 0-91. We are grateful for bulk dequeue support in AMQP 0-91, but in
> subsequent testing, we encountered other incompatibilities due to our use
> case. (Unfortunately have not had time to dig further). Given that the
> v0.16
> band-aid solution eliminated the immediate urgency, we feel that taking the
> time to build a solution around AMQP 0-10 is the right approach going
> forward.
>
> By the way, what is the typical release time frame for Qpid (with
> v0.32/trunk in mind)?
> Cheers, Dan
>
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Re-1-Queue-with-2-Consumers-turn-off-pre-fetching-tp6934582p7613381.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>

Re: 1 Queue with 2 Consumers - turn off pre-fetching?

[Rob Godfrey <ro...@gmail.com>]

Hi Helen,

glad to hear you have things working.

The single consumer against multiple queues code is already committed to
trunk, and so will be in 0.32 ... I've not had the chance to do a huge
amount of testing yet, so you might want to play with it a bit before 0.32
gets finalised.  I'm not sure what the release plan for 0.32 is yet - I'd
be quite keen to get another Java release out by the end of the year.


To enable the feature on the broker you currently need to run with
-Dqpid.enableMultiQueueConsumers=true (or you can set it directly as a
"context variable" [1] in the broker configuration).

To consume from such an address you need to use an address string like


ADDR: '' ; {node : { type : queue }, link : { x-subscribe : { arguments : {
x-multiqueue : [ q1, q2, q3 ] } } } }

where q1, q2 and q3 are the queues you wish to consumer from.


If you do get a chance to do any testing and find any issues (or if you
need any help with the upgrading of the configuration of your current
broker to what is now on trunk), please let me know,

Cheers,
Rob


[1]
https://qpid.apache.org/releases/qpid-trunk/java-broker/book/Java-Broker-Management-Managing-Broker.html#Java-Broker-Management-Managing-Broker-Context

On 21 October 2014 00:30, Helen Kwong <he...@gmail.com> wrote:

> Hi Rob,
>
> We have implemented the workaround Dan mentioned before -- by suspending
> the session with sendSuspendChannel() and releasing prefetched messages
> with rejectMessagesForConsumerTag() whenever we detect a long-running
> message -- and so far it is working as expected. Thank you again for all
> your help.
>
> For the long term solution, we plan to upgrade our client and broker to the
> latest version, and use the "single consumer across many queues" feature
> that you're building. So I'm wondering:
> 1. Do you still plan to have this available in v0.32 of Qpid?
> 2. What is the planned release date of v0.32?
>
> Thanks!
>
> Helen
>
> On Tue, Sep 9, 2014 at 12:54 PM, xiaodan.wang <xiaodan.wang@salesforce.com
> >
> wrote:
>
> > Hi Rob, in case you are interested, wanted to give you an update from our
> > side. First, let me begin by saying how awesome the Qpid community has
> > been.
> > Your patience and responsiveness in addressing our questions is way
> beyond
> > what we could have hoped for from this community. I think I speak for
> > Salesforce in saying that we owe you a couple rounds of beer, in case you
> > are ever bored and in San Francisco :)
> >
> > Our plan is to address per-consumer prefetching on the v0.16 clients we
> > currently run in production using the suspend & release prefetched
> messages
> > approach described above. The general flow of our solution is as follows:
> >
> > 1) Each time a message is dispatched and onMessage is invoked, we
> register
> > the parent Qpid session with a session tracker
> >
> > 2)  The session tracker is scheduled every x seconds and looks for Qpid
> > sessions that is processing a long running messages
> >
> > 3) Once the session tracker finds a long running message, it invokes
> > "sendSuspendChannel" and "rejectMessagesForConsumerTag" on the session to
> > suspend dequeue and release any prefetched messages
> >
> > 4) When the long running message is completed, it unregisters the parent
> > Qpid session from the session tracker and invokes "sendSuspendChannel" to
> > resume dequeue
> >
> > We are in the process of performance testing this solution to make sure
> > that
> > a) overhead of suspend and release is manageable and b) no anomalies
> arise
> > from repeatedly suspending and unsuspending the session
> >
> >
> > For our long term plan, we plan to upgrade to the latest version of the
> > client/broker and use the single-consumer-multiple-queue feature you've
> > built in v0.32 for AMQP 0-10. After we've patched our v0.16 client, we
> plan
> > to focus on vetting this.
> >
> > We are also abandoning efforts to switch to per-session prefetch with
> AMQP
> > 0-91. We are grateful for bulk dequeue support in AMQP 0-91, but in
> > subsequent testing, we encountered other incompatibilities due to our use
> > case. (Unfortunately have not had time to dig further). Given that the
> > v0.16
> > band-aid solution eliminated the immediate urgency, we feel that taking
> the
> > time to build a solution around AMQP 0-10 is the right approach going
> > forward.
> >
> > By the way, what is the typical release time frame for Qpid (with
> > v0.32/trunk in mind)?
> > Cheers, Dan
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://qpid.2158936.n2.nabble.com/Re-1-Queue-with-2-Consumers-turn-off-pre-fetching-tp6934582p7613381.html
> > Sent from the Apache Qpid users mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: users-help@qpid.apache.org
> >
> >
>