You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by gsupa <gr...@csx.com> on 2018/11/12 20:07:27 UTC
Python Proton - Multiple consumers on dynamic queue
Hi All,
I'm working on migrating a python2 app to python3, which also means
transitioning to Proton. The application is communicating with a c++ qpid
broker.
*One of the last roadblocks I'm having in this transition is figuring out
how to dynamically generate a queue with bindings hooked up.* The current
application will create a queue with x-declare and x-bindings, and can scale
with more consumers. These new consumers will connect to this existing queue
(or create if it does not exist) and messages on this queue will be
distributed across these consumers. Additionally, if there are no consumers,
the queue will be automatically deleted, and messages sent to the exchange
will be routed to an alternate exchange.
Old usage:
{ create: always, node: { type: queue, durable: False, x-declare: {
exclusive: False, auto-delete: True, alternate-exchange: e_fed_example },
x-bindings: [ { exchange: 'e_example' } ] }, link: { durable: False } }
(The actual exchange e_example also has alternate-exchange defined)
Some possible solutions I've evaluated:
1) I could dynamically create a receiver on the exchange for each consumer,
but this would mean messages duplicated to all consumers, instead of being
distributed.
2) The broker has an option for queues with a certain pattern to be created
on demand. This allows our consumers to connect to the same queue, and the
queue will auto-delete if all consumers disconnect (allowing for the
alternate exchange on the exchange to kick in). However, the queue is not
bound to an exchange, which isn't helpful.
3) I've seen a c++ example which utilizes 'shared' 'capabilities' when
creating a connection which would theoretically allow new receivers to
connect to the same queue, but I haven't seen any documentation on this for
Python.
All advice is greatly appreciated.
Thanks,
Griffin
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Python Proton - Multiple consumers on dynamic queue
Posted by gsupa <gr...@csx.com>.
Gordon Sim wrote
> I'm sorry, I gave you a broken example! I have pushed a fix:
> https://github.com/grs/amqp_subscriptions/commit/82d1fdebc4c15d53d5493dbc67f3f3f4773f8fe9
> can you try again with that?
>
> The problem was that the capabilities were being encoded as a list and
> not an array, so the broker was ignoring them.
This appears to be working! Thank you so much!
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Python Proton - Multiple consumers on dynamic queue
Posted by Gordon Sim <gs...@redhat.com>.
On 14/11/18 15:54, gsupa wrote:
> Gordon Sim wrote
>> We should only have one queue when the shared capability is requested.
>> What version of the broker are you using? Can you try with just the
>> 'shared' capability (no 'global')?
>
> Yea, I'm seeing the same result using Capabilities(['shared'] and
> Capabilities(['global', 'shared'])
I'm sorry, I gave you a broken example! I have pushed a fix:
https://github.com/grs/amqp_subscriptions/commit/82d1fdebc4c15d53d5493dbc67f3f3f4773f8fe9
can you try again with that?
The problem was that the capabilities were being encoded as a list and
not an array, so the broker was ignoring them.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Python Proton - Multiple consumers on dynamic queue
Posted by gsupa <gr...@csx.com>.
Gordon Sim wrote
> We should only have one queue when the shared capability is requested.
> What version of the broker are you using? Can you try with just the
> 'shared' capability (no 'global')?
Yea, I'm seeing the same result using Capabilities(['shared'] and
Capabilities(['global', 'shared'])
The version of qpidd is:
qpidd (qpid-cpp) version 1.36.0
Is there a way to get trace data that could further help debugging?
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Python Proton - Multiple consumers on dynamic queue
Posted by Gordon Sim <gs...@redhat.com>.
On 13/11/18 20:31, gsupa wrote:
> Gordon Sim wrote
>> You can also define a 'topic', which is basically an exchange and the
>> queue settings to be used for subscriptions to it.
>
> Yea, I've gotten this to execute but it falls to the same issue I have with
> creating a receiver on an exchange. I want multiple receivers to be able to
> connect, but for every matching message *only one receiver should get it*.
> Instead, when I've created receivers on a topic/exchange, all receivers will
> create their own dynamically generated queue and each will receive the
> message. Is there a queue property/setting that I'm missing?
That *should* be handled by the 'shared' capability.
> Gordon Sim wrote
>> https://github.com/grs/amqp_subscriptions/blob/master/e.py#L41 is an
>> example of specifying a shared capability (it uses a small utility class
>> which is also in that repo).
>
> I experience the same situation above when using Capabilities. Here's my set
> up.
>
> I place the following line in the on_start():
>
> event.container.create_receiver("localhost:5672/example_exchange",
> name='test', options=[Capabilities(['global', 'shared'])])
>
> I run the file twice, and qpid-stat -q yields:
> queue dur autoDel excl msg msgIn
> msgOut bytes bytesIn bytesOut cons bind
> ==========================================================================================================================
> 05aaff7a-4fc7-4ab6-935d-7f2ba8504d99_test Y Y 0 0
> 0 0 0 0 1 2
> a5a958f7-ee55-4e75-8d35-f7e2b9968a35_test Y Y 0 0
> 0 0 0 0 1 2
We should only have one queue when the shared capability is requested.
What version of the broker are you using? Can you try with just the
'shared' capability (no 'global')?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Python Proton - Multiple consumers on dynamic queue
Posted by gsupa <gr...@csx.com>.
Gordon Sim wrote
> You can also define a 'topic', which is basically an exchange and the
> queue settings to be used for subscriptions to it.
Yea, I've gotten this to execute but it falls to the same issue I have with
creating a receiver on an exchange. I want multiple receivers to be able to
connect, but for every matching message *only one receiver should get it*.
Instead, when I've created receivers on a topic/exchange, all receivers will
create their own dynamically generated queue and each will receive the
message. Is there a queue property/setting that I'm missing?
Gordon Sim wrote
> https://github.com/grs/amqp_subscriptions/blob/master/e.py#L41 is an
> example of specifying a shared capability (it uses a small utility class
> which is also in that repo).
I experience the same situation above when using Capabilities. Here's my set
up.
I place the following line in the on_start():
event.container.create_receiver("localhost:5672/example_exchange",
name='test', options=[Capabilities(['global', 'shared'])])
I run the file twice, and qpid-stat -q yields:
queue dur autoDel excl msg msgIn
msgOut bytes bytesIn bytesOut cons bind
==========================================================================================================================
05aaff7a-4fc7-4ab6-935d-7f2ba8504d99_test Y Y 0 0
0 0 0 0 1 2
a5a958f7-ee55-4e75-8d35-f7e2b9968a35_test Y Y 0 0
0 0 0 0 1 2
When sending a message to example_exchange, both receivers receive the
message, when I expect only one to receive the message.
Again, thank you for your help.
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Python Proton - Multiple consumers on dynamic queue
Posted by Gordon Sim <gs...@redhat.com>.
On 12/11/18 20:07, gsupa wrote:
> Hi All,
>
> I'm working on migrating a python2 app to python3, which also means
> transitioning to Proton. The application is communicating with a c++ qpid
> broker.
>
> *One of the last roadblocks I'm having in this transition is figuring out
> how to dynamically generate a queue with bindings hooked up.* The current
> application will create a queue with x-declare and x-bindings, and can scale
> with more consumers. These new consumers will connect to this existing queue
> (or create if it does not exist) and messages on this queue will be
> distributed across these consumers. Additionally, if there are no consumers,
> the queue will be automatically deleted, and messages sent to the exchange
> will be routed to an alternate exchange.
>
> Old usage:
> { create: always, node: { type: queue, durable: False, x-declare: {
> exclusive: False, auto-delete: True, alternate-exchange: e_fed_example },
> x-bindings: [ { exchange: 'e_example' } ] }, link: { durable: False } }
>
> (The actual exchange e_example also has alternate-exchange defined)
>
> Some possible solutions I've evaluated:
> 1) I could dynamically create a receiver on the exchange for each consumer,
> but this would mean messages duplicated to all consumers, instead of being
> distributed.
> 2) The broker has an option for queues with a certain pattern to be created
> on demand. This allows our consumers to connect to the same queue, and the
> queue will auto-delete if all consumers disconnect (allowing for the
> alternate exchange on the exchange to kick in). However, the queue is not
> bound to an exchange, which isn't helpful.
You can also define a 'topic', which is basically an exchange and the
queue settings to be used for subscriptions to it.
> 3) I've seen a c++ example which utilizes 'shared' 'capabilities' when
> creating a connection which would theoretically allow new receivers to
> connect to the same queue, but I haven't seen any documentation on this for
> Python.
https://github.com/grs/amqp_subscriptions/blob/master/e.py#L41 is an
example of specifying a shared capability (it uses a small utility class
which is also in that repo).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org