qpid for multiple producers and consumers

[Ramashish Baranwal <ra...@gmail.com>]

Hi,

I am trying to evaluate qpid for an application with distributed
producers and consumers. The application has multiple producers and
multiple consumers, all subscribed to the same queue. The requirement
is that any consumer should be able to receive a message and process
it. The message processing takes some time, so while a consumer is
processing a message other consumers should be able to retrieve other
messages present in the queue so that the processing can happen in
parallel by multiple consumers.

It seems this should be easy to do with qpid, but I haven't been able
to figure out till now. I tried the python examples, direct
producer/consumer looked like what I wanted till I saw that while one
consumer is fetching a message, the other consumer is not able to.
When I close the first consumer, the other consumer picks up from
where the first consumer left. What I want is a parallel behaviour as
mentioned above.

I probably have overlooked something. Any ideas, or suggestions?

Thanks in advance :)
Ram

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

Re: qpid for multiple producers and consumers

[Gordon Sim <gs...@redhat.com>]

On 02/16/2010 04:31 PM, Ramashish Baranwal wrote:
>>
>> Try setting flow control such that each consumer only gets one message at a
>> time. I.e. instead of calling start() on the incoming queue (which gives
>> unlimited credit) allocate credit yourself. (Attached patch attempts to
>> describe that more precisely).
>>
>> --Gordon.
>>
>
> Thanks Gordon! That worked. So it seems the flow control will make the
> client receive one message at a time, right? However, after receiving
> just one message, the queue.get() is timing out. Do I need to do
> something else to continue receiving more messages one at a time?

Yes, you can 'move' the credit window by sending completions. There may 
now be a more elegant way of doing this (Rafi?), but the following will 
work:

@@ -83,6 +85,8 @@
  while content != final:
  	message = queue.get(timeout=10)
  	content = message.body
+        session.receiver._completed.add(message.id)
+        session.channel.session_completed(session.receiver._completed)
          session.message_accept(RangedSet(message.id))
  	print content

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

Re: qpid for multiple producers and consumers

[Ramashish Baranwal <ra...@gmail.com>]

>
> Try setting flow control such that each consumer only gets one message at a
> time. I.e. instead of calling start() on the incoming queue (which gives
> unlimited credit) allocate credit yourself. (Attached patch attempts to
> describe that more precisely).
>
> --Gordon.
>

Thanks Gordon! That worked. So it seems the flow control will make the
client receive one message at a time, right? However, after receiving
just one message, the queue.get() is timing out. Do I need to do
something else to continue receiving more messages one at a time?

Regards,
Ramashish

---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project:      http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org

Re: qpid for multiple producers and consumers

[Gordon Sim <gs...@redhat.com>]

On 02/11/2010 07:01 AM, Ramashish Baranwal wrote:
> Hi,
>
> I am trying to evaluate qpid for an application with distributed
> producers and consumers. The application has multiple producers and
> multiple consumers, all subscribed to the same queue. The requirement
> is that any consumer should be able to receive a message and process
> it. The message processing takes some time, so while a consumer is
> processing a message other consumers should be able to retrieve other
> messages present in the queue so that the processing can happen in
> parallel by multiple consumers.
>
> It seems this should be easy to do with qpid, but I haven't been able
> to figure out till now. I tried the python examples, direct
> producer/consumer looked like what I wanted till I saw that while one
> consumer is fetching a message, the other consumer is not able to.
> When I close the first consumer, the other consumer picks up from
> where the first consumer left. What I want is a parallel behaviour as
> mentioned above.
>
> I probably have overlooked something. Any ideas, or suggestions?

Try setting flow control such that each consumer only gets one message 
at a time. I.e. instead of calling start() on the incoming queue (which 
gives unlimited credit) allocate credit yourself. (Attached patch 
attempts to describe that more precisely).

--Gordon.