multiple consumers on one or more queues in parallel

[Matjaž Ostroveršnik <Ma...@halcom.si>]

Hi all,

Thanks all for your fast answers on my previous question. 

I am trying to make a client / consumer who will attach to one or more queues and read the messages from the queues and process them.
Since processing of a single message can take a while I'd like to have multiple consumers running in parallel and consuming the messages as soon they arrive.

I've noticed that even if I run several clients (see below) in parallel, only one is getting all the messages from the queue. Expirementation with acknowledgements and capacity did not help.
Question:

How can I read from several queues at the same time and in parallel with several clients?


Thanks in advance

Matjaž

The relevant fraction of code is listed below:

        int delay = 60;
          if (m_session.nextReceiver(rec, delay * qmsg::Duration::SECOND)) //FIXME make it configurable

          {
            Log::info(__S("Started receiving messages from the queue: '") + rec.getName() + "'");
            do
            {
              delay = 1;
              one_batch_of_files = read_from_queue_and_save_to_disk(rec, file_dscr);
              files_written += one_batch_of_files;
              Log::info(
                  __S(one_batch_of_files) + " files taken from the queue: '" + rec.getName()
                      + "' and written to disk. Still to process:" + __S(m_session.getReceivable()));
            }
            while (m_session.nextReceiver(rec, delay * qmsg::Duration::SECOND) && //FIXME make it configurable
                process::exit_on_signal());
          }
          else
          {
            Log::info(__S("No receiver with messages. Waiting...")+__S(delay));
          }
        Log::info("Files left to process:" +__S(m_session.getReceivable()));


My queue definition is as follows:
max_msg_cnt=1000000
que_size_pages=600000000
jnl_files=64
fs=850
param="--max-queue-size=$que_size_pages --file-size=$fs --file-count=$jnl_files --max-queue-count=$max_msg_cnt --limit-policy=flow-to-disk --durable --group-header=\"filename\" --shared-groups "
qpid-config add queue Alpha   $param
qpid-config add queue Beta  $param
qpid-config add queue Gamma $param
qpid-config add queue Delta $param

Re: multiple consumers on one or more queues in parallel

[Gordon Sim <gs...@redhat.com>]

On 11/19/2012 10:42 AM, Gordon Sim wrote:
> On 11/18/2012 02:45 PM, Matjaž Ostroveršnik wrote:
>> I've noticed that even if I run several clients (see below) in
>> parallel, only one is getting all the messages from the queue.
>> Expirementation with acknowledgements and capacity did not help.
>
> What did you set the capacity to? If you set it to 0, does that not help?

After sending this I saw your queue configuration used shared message 
groups. The definition of that feature is that messages for the same 
group are dispatched such that they can be guaranteed to be processed in 
order. Specifically this means that while there are unacknowledged 
messages outstanding for that group, other message in the same group 
will be sent to the consumer to whom the outstanding messages were sent.

So, how are you using message groups? In your tests did you have 
messages from many different groups being sent?

My advice would still be to set capacity to 0 to void the batching that 
prefetch tends to lead to (which in small flows can appear as if one 
consumer gets everything); acknowledgements will also have an impact - 
if you acknowledge each message then the association between a group and 
a consume will have the shortest possible scope.

Note that these two configurations (capacity = 0 and acknowledgement on 
every message) will reduce the throughput per connection. However that 
may be less important than getting a smoother balancing between consumers.

If you have already tried that and it doesn't work, there would be 
something not behaving as expected. In that event can you include a 
complete example, or at least the details of the creation of the 
receiver and the pattern of sending messages?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org

Re: multiple consumers on one or more queues in parallel

[Gordon Sim <gs...@redhat.com>]

On 11/18/2012 02:45 PM, Matjaž Ostroveršnik wrote:
> I've noticed that even if I run several clients (see below) in parallel, only one is getting all the messages from the queue. Expirementation with acknowledgements and capacity did not help.

What did you set the capacity to? If you set it to 0, does that not help?

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org