You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by "Wisler, Trina" <Tr...@hp.com> on 2011/10/05 17:07:33 UTC
ring queues
Hi,
I have a queue setup with max message count of 100 and as a ring queue. The single client creating that queue will consume 1 message and then wait indefinitely (for the purpose of my test). I am producing a total of 250 messages. As I would expect, qpid-tool reports that 250 messages have been enqueued and 150 have been dequeued (presumably due to the oldest messages being overwritten). Here is a snippet from qpid-tool
property arguments {u'qpid.max_count': 100, u'qpid.policy_type': u'ring'}
property altExchange <NULL>
statistic msgTotalEnqueues 250 messages
statistic msgTotalDequeues 150
statistic msgTxnEnqueues 0
statistic msgTxnDequeues 0
statistic msgPersistEnqueues 0
statistic msgPersistDequeues 0
statistic msgDepth 100
However, when I resume my client, I would expect that it only will receive the 100 remaining messages on the queue, but instead it completes with all 250. I'm slightly confused by this because the definition of a ring queue states : ""RING : start overwriting messages in a ring based on sizing. If head meets tail, advance head". The behavior I want is for it to overwrite older messages once the queue reaches max size.
We are using qpid .06, working on upgrading to .12.
Thanks in advance.
Trina Wisler
RE: ring queues
Posted by "Wisler, Trina" <Tr...@hp.com>.
This sounds like what we want. I did a quick test using what you suggested below and it worked as expected.
Thank you again for your time.
Trina
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: Wednesday, November 02, 2011 9:10 AM
To: users@qpid.apache.org
Subject: Re: ring queues
On 11/02/2011 03:53 PM, Wisler, Trina wrote:
> That is very interesting. I was not aware of this behavior. We are using all default setting for everything other than trying to implement the ring queues for the declareQueue call (again, very simple example just trying to test behavior).
>
> Thank you for your replies. I am interested in this buffering behavior in terms of buffering limits (how many messages get buffered, if there are setting that I can change that would affect this, etc.).
If you are able to switch to the qpid::messaging API, that is in my view
much simpler and more powerful. (For an overview have a look at the
guide: http://qpid.apache.org/books/0.12/Programming-In-Apache-Qpid/).
However, even in the qpid::client API you can control the in-flow of
messages for your subscription.
Have a look at SubscriptionSettings, and instance of which can be passed
to the SubscriptionManager::subscribe() call. You can set a prefetch
window, and this will restrict the number of messages that the broker
will deliver ahead of those that you actually process.
E.g.
subMgr.subscribe(listener, "my-queue",
SubscriptionSettings(FlowControl::messageWindow(10)));
which would set a prefetch of 10 messages.
In your experiment this would still mean you could receive up to twenty
messages as 10 could be prefetched while you were not processing them.
If you need to not receive any without cancelling the subscription you
can start with a window of 0 or you can call Session::messageStop()
passing in the name of your subscription (Subscription::getName()). You
can use the Subscription::setFlowControl() to reopen the window.
Hope this helps.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: ring queues
Posted by Gordon Sim <gs...@redhat.com>.
On 11/02/2011 03:53 PM, Wisler, Trina wrote:
> That is very interesting. I was not aware of this behavior. We are using all default setting for everything other than trying to implement the ring queues for the declareQueue call (again, very simple example just trying to test behavior).
>
> Thank you for your replies. I am interested in this buffering behavior in terms of buffering limits (how many messages get buffered, if there are setting that I can change that would affect this, etc.).
If you are able to switch to the qpid::messaging API, that is in my view
much simpler and more powerful. (For an overview have a look at the
guide: http://qpid.apache.org/books/0.12/Programming-In-Apache-Qpid/).
However, even in the qpid::client API you can control the in-flow of
messages for your subscription.
Have a look at SubscriptionSettings, and instance of which can be passed
to the SubscriptionManager::subscribe() call. You can set a prefetch
window, and this will restrict the number of messages that the broker
will deliver ahead of those that you actually process.
E.g.
subMgr.subscribe(listener, "my-queue",
SubscriptionSettings(FlowControl::messageWindow(10)));
which would set a prefetch of 10 messages.
In your experiment this would still mean you could receive up to twenty
messages as 10 could be prefetched while you were not processing them.
If you need to not receive any without cancelling the subscription you
can start with a window of 0 or you can call Session::messageStop()
passing in the name of your subscription (Subscription::getName()). You
can use the Subscription::setFlowControl() to reopen the window.
Hope this helps.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: ring queues
Posted by "Wisler, Trina" <Tr...@hp.com>.
That is very interesting. I was not aware of this behavior. We are using all default setting for everything other than trying to implement the ring queues for the declareQueue call (again, very simple example just trying to test behavior).
Thank you for your replies. I am interested in this buffering behavior in terms of buffering limits (how many messages get buffered, if there are setting that I can change that would affect this, etc.).
Thanks again!
Trina
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: Wednesday, November 02, 2011 8:31 AM
To: users@qpid.apache.org
Subject: Re: ring queues
On 11/02/2011 03:23 PM, Wisler, Trina wrote:
> Yes, the client was subscribed to the queue I was simply preventing
> it from processing messages. Basically my call to listen would be
> delayed until I was ready for it to start consuming - but it had
> already called declareQueue and subscribeQueue, etc..
Ok, that is likely the issue. What SubscriptionSettings did you use?
The default settings gives the broker unlimited credit which means it
can send as many messages as it can. Ss the messages are published they
are delivered to the client and queued up by the library waiting to be
processed. The fact that they are then dequeued by subsequent
publications doesn't alter this.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: ring queues
Posted by Gordon Sim <gs...@redhat.com>.
On 11/02/2011 03:23 PM, Wisler, Trina wrote:
> Yes, the client was subscribed to the queue I was simply preventing
> it from processing messages. Basically my call to listen would be
> delayed until I was ready for it to start consuming - but it had
> already called declareQueue and subscribeQueue, etc..
Ok, that is likely the issue. What SubscriptionSettings did you use?
The default settings gives the broker unlimited credit which means it
can send as many messages as it can. Ss the messages are published they
are delivered to the client and queued up by the library waiting to be
processed. The fact that they are then dequeued by subsequent
publications doesn't alter this.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: ring queues
Posted by "Wisler, Trina" <Tr...@hp.com>.
Yes, the client was subscribed to the queue I was simply preventing it from processing messages. Basically my call to listen would be delayed until I was ready for it to start consuming - but it had already called declareQueue and subscribeQueue, etc..
Trina
-----Original Message-----
From: Gordon Sim [mailto:gsim@redhat.com]
Sent: Wednesday, November 02, 2011 2:34 AM
To: users@qpid.apache.org
Subject: Re: ring queues
On 11/01/2011 10:27 PM, Wisler, Trina wrote:
> I created a queue size of 100. I produced 54875 messages, backed up
> the queue and didn't allow my client to consume any. Once I was done
> producing message, I allowed my client to consume and my client
> received 2384 messages immediately.
When you say you didn't allow your client to consume any messages, was
the client subscribed to the queue at the point the messages were
published? I.e. do you mean simply that you prevented any processing of
messages by your application?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: ring queues
Posted by Gordon Sim <gs...@redhat.com>.
On 11/01/2011 10:27 PM, Wisler, Trina wrote:
> I created a queue size of 100. I produced 54875 messages, backed up
> the queue and didn't allow my client to consume any. Once I was done
> producing message, I allowed my client to consume and my client
> received 2384 messages immediately.
When you say you didn't allow your client to consume any messages, was
the client subscribed to the queue at the point the messages were
published? I.e. do you mean simply that you prevented any processing of
messages by your application?
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
RE: ring queues
Posted by "Wisler, Trina" <Tr...@hp.com>.
Thanks for your reply (yes it was a while ago). We finally upgraded to qpid .12 and we are seeing the same behavior.
My test setup:
** C++ brokers, C++ clients
** Providers that publish simple messages
** Client will not consume any messages until I hit the "go" button essentially as I am wanting the queue to fill up and overwrite itself for a while.
If I have a queue size of 10 (for simplicity), and I back up the queue to where it starts overwriting (I write about 1000 messages), and then let my client start consuming once all messages have been published - it should only read 10 messages. But instead, it always reads between 380 and 397. I find this highly odd. Here is what qpid-tool shows for a few of the queue attributes.
arguments {u'qpid.max_count': 10, u'qpid.policy_type': u'ring'}
msgDepth 10
I am using the C++ API as show below.
void qpid::client::QueueOptions::setSizePolicy
( QueueSizePolicy sp,
uint64_t maxSize,
uint32_t maxCount
)
The documented qpid example is as such:
_______________
Create a queue that will support 1000 messages into a RING buffer
#include "qpid/client/QueueOptions.h"
QueueOptions qo;
qo.setSizePolicy(RING,0,1000);
session.queueDeclare(arg::queue=queue, arg::arguments=qo);
________________
My sample client snippet of code is this (modeling mostly the above example):
qpid::client::QueueOptions qo;
qo.setSizePolicy(qpid::client::RING,0,10);
session->queueDeclare(client::arg::queue=queue, client::arg::exclusive=false,
client::arg::autoDelete=true ,client::arg::arguments=qo);
So, back to my example, why do I receive more than 10 messages?
I did another quick test to see what would happen if I increased my queue size. I created a queue size of 100. I produced 54875 messages, backed up the queue and didn't allow my client to consume any. Once I was done producing message, I allowed my client to consume and my client received 2384 messages immediately.
This is what qpid-tool showed at the moment before I finally allowed my client to consume (you can see there is only 100 messages on the queue):
arguments {u'qpid.max_count': 100, u'qpid.policy_type': u'ring'}
msgTotalEnqueues 54875
msgTotalDequeues 54775 -->these dequeues are due to the ring queue overwrites
msgDepth 100
I do not understand this behavior.
All we want is to have a ring queue of size X and we expect that if we hit X then older messages will be overwritten. Period.
Thanks in advance for your time,
Trina Wisler
-----Original Message-----
From: Carl Trieloff [mailto:cctrieloff@redhat.com]
Sent: Wednesday, October 05, 2011 9:29 AM
To: users@qpid.apache.org
Subject: Re: ring queues
On 10/05/2011 11:07 AM, Wisler, Trina wrote:
> Hi,
>
> I have a queue setup with max message count of 100 and as a ring queue. The single client creating that queue will consume 1 message and then wait indefinitely (for the purpose of my test). I am producing a total of 250 messages. As I would expect, qpid-tool reports that 250 messages have been enqueued and 150 have been dequeued (presumably due to the oldest messages being overwritten). Here is a snippet from qpid-tool
>
> property arguments {u'qpid.max_count': 100, u'qpid.policy_type': u'ring'}
> property altExchange <NULL>
> statistic msgTotalEnqueues 250 messages
> statistic msgTotalDequeues 150
> statistic msgTxnEnqueues 0
> statistic msgTxnDequeues 0
> statistic msgPersistEnqueues 0
> statistic msgPersistDequeues 0
> statistic msgDepth 100
>
> However, when I resume my client, I would expect that it only will receive the 100 remaining messages on the queue, but instead it completes with all 250. I'm slightly confused by this because the definition of a ring queue states : ""RING : start overwriting messages in a ring based on sizing. If head meets tail, advance head". The behavior I want is for it to overwrite older messages once the queue reaches max size.
>
> We are using qpid .06, working on upgrading to .12.
>
> Thanks in advance.
> Trina Wisler
>
The behaviour you see seems incorrect. The ring queue implementation has
been refactored between 0.6 and 0.12 so if you are going to update, if
you can do that and report back, this make is easier to try work out
what is going on if we are both on the latest version.
btw, which client are you using, not that it should matter.
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: ring queues
Posted by Carl Trieloff <cc...@redhat.com>.
On 10/05/2011 11:07 AM, Wisler, Trina wrote:
> Hi,
>
> I have a queue setup with max message count of 100 and as a ring queue. The single client creating that queue will consume 1 message and then wait indefinitely (for the purpose of my test). I am producing a total of 250 messages. As I would expect, qpid-tool reports that 250 messages have been enqueued and 150 have been dequeued (presumably due to the oldest messages being overwritten). Here is a snippet from qpid-tool
>
> property arguments {u'qpid.max_count': 100, u'qpid.policy_type': u'ring'}
> property altExchange <NULL>
> statistic msgTotalEnqueues 250 messages
> statistic msgTotalDequeues 150
> statistic msgTxnEnqueues 0
> statistic msgTxnDequeues 0
> statistic msgPersistEnqueues 0
> statistic msgPersistDequeues 0
> statistic msgDepth 100
>
> However, when I resume my client, I would expect that it only will receive the 100 remaining messages on the queue, but instead it completes with all 250. I'm slightly confused by this because the definition of a ring queue states : ""RING : start overwriting messages in a ring based on sizing. If head meets tail, advance head". The behavior I want is for it to overwrite older messages once the queue reaches max size.
>
> We are using qpid .06, working on upgrading to .12.
>
> Thanks in advance.
> Trina Wisler
>
The behaviour you see seems incorrect. The ring queue implementation has
been refactored between 0.6 and 0.12 so if you are going to update, if
you can do that and report back, this make is easier to try work out
what is going on if we are both on the latest version.
btw, which client are you using, not that it should matter.
Carl.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org