You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by fadams <fr...@blueyonder.co.uk> on 2011/03/10 18:34:04 UTC
Is it possible to have persistent circular queues
I know that it's possible to have in memory circular queues by setting the
behaviour to "ring" and it's clearly possible to have persistent/durable
queues but is it possible to make a durable queue circular?
Under the hood the persistence layer is clearly a circular buffer, but it
doesn't appear to be possible to set it up such that newest data actually
overwrites oldest.
This would be pretty useful from my perspective as I'd like a queue larger
than my available memory, but I'd also like to silently dump the oldest
stuff without throwing an exception back to the producer.
Is this possible?
Any neat workarounds?
Is this a feature that others are interested in so likely to be in the
pipeline?
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Is-it-possible-to-have-persistent-circular-queues-tp6158613p6158613.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to have persistent circular queues
Posted by Gordon Sim <gs...@redhat.com>.
On 03/18/2011 05:54 PM, fadams wrote:
> Hi again,
> This still seems to be blowing up on me I'm afraid....
>
> Given the last bit of advice from Gordon Sim (thanks for all your help so
> far Gordon!!) I tried:
>
> qpidd --default-queue-limit 0
>
> Then I added my queue:
>
> qpid-config add queue myqueue --durable --max-queue-count 50000 --file-size
> 5000 --file-count 16 --limit-policy ring
>
> to create a queue of roughly 5GB
>
> This seemed to be added OK, but when I fired up my producer (with no
> consumers enabled) I get an exception thrown almost instantly (I guess on
> the first flush)
>
> qpidd says:
>
> error Execution exception: resource-limit-exceeded: Policy exceeded on
> myqueue, policy: size: unlimited; count: max=50000, current=0; type=ring
> (qpid/broker/QueuePolicy.cpp:86)
Sorry! That is indeed another bug
(https://issues.apache.org/jira/browse/QPID-3180). I've committed a fix
to trunk for that. Apologies for the unhelpful advice I gave previously
and thanks for your patience and perseverance here.
> It looks very odd that qpidd says "Policy exceeded" and also "policy: size:
> unlimited"
>
>
>
> My (Java) producer says:
>
> ItemProducer onException
> ItemProducer: exception: Invalid Session
> javax.jms.IllegalStateException: Invalid Session
> at
> org.apache.qpid.client.BasicMessageProducer.checkPreConditions(BasicMessageProducer.java:550)
> at
> org.apache.qpid.client.BasicMessageProducer.send(BasicMessageProducer.java:277)
>
> After 12 messages.
>
>
>
> I'd love to know what I'm doing wrong and suggestions for qpid-config
> options to get me a circular queue larger than available memory would be
> very welcome indeed.
That isn't possible at present. What we really want for handling queues
larger than the available memory is a proper paging solution. All we
have at present however is a cheap hack to reuse the policy mechanism
('flow to disk') and release message content from memory once the limit
is reached. That means you can't combine the ring and flow-to-disk
policies and also makes the handling of queues larger than memory quite
slow. It also only really helps at all for large messages; a message
'handle' (including all headers) is still kept in memory for all messages.
> The reason that I care about this is that I've got a real time producer
> client on an operational system and I'm using qpid to work rather like a
> data mart, with consumers able to connect in and suck up the stuff my
> producer is sending. What I'd really like to avoid is the case where a
> consumer client fails (or is too slow) which then throws an exception back
> to the producer - the producer may be providing data to dozens of consumers
> and doesn't especially care one of them fails (that's their problem!!) OTOH
> I would like to provide a reasonable level of buffering for consumers to
> allow for reasonable consumer outages (hence why I'd like to be able to
> support queues larger than my available memory).
>
> It would be really bad in my system for a failing consumer to kill my
> producer. The producer is pretty well written to handle exceptions and
> reconnect, but with high rate data and dozens of consumers I'm not keen to
> have a failing consumer affect everything else on a high value critical
> operational system.
>
> I'd really appreciate any advice for handling this scenario - I'd have
> thought that this would be a fairly common sort of pattern??
Yes, it is and the ring queue policy is currently the solution. Its just
that the support for very large queues (larger than available memory) is
not really adequate and is not orthogonal to the solution for ring queues.
Does the data published become irrelevant in any way? E.g. does it
expire after some time (TTL may be useful) or do later updates
invalidate earlier messages (where perhaps an LVQ might help)...
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to have persistent circular queues
Posted by fadams <fr...@blueyonder.co.uk>.
Hi again,
This still seems to be blowing up on me I'm afraid....
Given the last bit of advice from Gordon Sim (thanks for all your help so
far Gordon!!) I tried:
qpidd --default-queue-limit 0
Then I added my queue:
qpid-config add queue myqueue --durable --max-queue-count 50000 --file-size
5000 --file-count 16 --limit-policy ring
to create a queue of roughly 5GB
This seemed to be added OK, but when I fired up my producer (with no
consumers enabled) I get an exception thrown almost instantly (I guess on
the first flush)
qpidd says:
error Execution exception: resource-limit-exceeded: Policy exceeded on
myqueue, policy: size: unlimited; count: max=50000, current=0; type=ring
(qpid/broker/QueuePolicy.cpp:86)
It looks very odd that qpidd says "Policy exceeded" and also "policy: size:
unlimited"
My (Java) producer says:
ItemProducer onException
ItemProducer: exception: Invalid Session
javax.jms.IllegalStateException: Invalid Session
at
org.apache.qpid.client.BasicMessageProducer.checkPreConditions(BasicMessageProducer.java:550)
at
org.apache.qpid.client.BasicMessageProducer.send(BasicMessageProducer.java:277)
After 12 messages.
I'd love to know what I'm doing wrong and suggestions for qpid-config
options to get me a circular queue larger than available memory would be
very welcome indeed.
The reason that I care about this is that I've got a real time producer
client on an operational system and I'm using qpid to work rather like a
data mart, with consumers able to connect in and suck up the stuff my
producer is sending. What I'd really like to avoid is the case where a
consumer client fails (or is too slow) which then throws an exception back
to the producer - the producer may be providing data to dozens of consumers
and doesn't especially care one of them fails (that's their problem!!) OTOH
I would like to provide a reasonable level of buffering for consumers to
allow for reasonable consumer outages (hence why I'd like to be able to
support queues larger than my available memory).
It would be really bad in my system for a failing consumer to kill my
producer. The producer is pretty well written to handle exceptions and
reconnect, but with high rate data and dozens of consumers I'm not keen to
have a failing consumer affect everything else on a high value critical
operational system.
I'd really appreciate any advice for handling this scenario - I'd have
thought that this would be a fairly common sort of pattern??
MTIA
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Is-it-possible-to-have-persistent-circular-queues-tp6158613p6185534.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to have persistent circular queues
Posted by Gordon Sim <gs...@redhat.com>.
On 03/13/2011 06:29 PM, fadams wrote:
> Hi Again
>
> Re the response:
>
>
> Gordon Sim wrote:
>>
>> When using qpid-config it would be:
>>
>> qpid-config add queue my-durable-ring-queue --limit-policy ring
>> --max-queue-count N --durable
>>
>>
>
> This still doesn't seem to be doing what I'd hoped.
>
> I created a queue as follows:
>
> qpid-config add queue myqueue --durable --max-queue-count 50000 --file-size
> 5000 --file-count 16 --limit-policy ring
>
> so this should have created a queue of size 5000*64K*16 ~5GB
>
> My producer is sending messages of 50K so with max-queue-count of 50000 I'd
> expect 50K * 50K to be used ~2.5GB
>
> However when I run my producer with no consumer it happily runs through with
> no exception (I had a produce run of 100000 items to see what happens) but
> here's the rub - when I fire up my consumer it only receives 2096 items not
> 50000. 2096*50K ~104800000 which is roughly the default-queue-limit
>
> What I really wanted to achieve is a queue larger than my available memory,
> but I'd also like to silently dump the oldest stuff without throwing an
> exception back to the producer
That you can't do at present. You can either have the flow to disk
policy that will release message content from memory once the limit is
reached or you can have a ring policy that deletes the oldest message(s)
at that point.
> - which leads back to my original
> question......
>
> So is it possible to set up a very large durable queue that behaves as a
> ring buffer (of roughly the size of the queue as opposed to the
> default-queue-limit size)
This part is possible. However the size of the ring buffer is configured
separately from the size of the journal. The simplest way around your
issue above is to set the default-queue-limit to 0. The queue will then
always have 50000 messages on it, whatever the size of those messages.
There are a couple of bugs I've just noticed and raised JIRAs for that
prevent either setting the specific queues size to be unlimited[1] via
qpid-config, or specifying a large enough size in your case[2].
[1] https://issues.apache.org/jira/browse/QPID-3141
[2] https://issues.apache.org/jira/browse/QPID-3140
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to have persistent circular queues
Posted by fadams <fr...@blueyonder.co.uk>.
Hi Again
Re the response:
Gordon Sim wrote:
>
> When using qpid-config it would be:
>
> qpid-config add queue my-durable-ring-queue --limit-policy ring
> --max-queue-count N --durable
>
>
This still doesn't seem to be doing what I'd hoped.
I created a queue as follows:
qpid-config add queue myqueue --durable --max-queue-count 50000 --file-size
5000 --file-count 16 --limit-policy ring
so this should have created a queue of size 5000*64K*16 ~5GB
My producer is sending messages of 50K so with max-queue-count of 50000 I'd
expect 50K * 50K to be used ~2.5GB
However when I run my producer with no consumer it happily runs through with
no exception (I had a produce run of 100000 items to see what happens) but
here's the rub - when I fire up my consumer it only receives 2096 items not
50000. 2096*50K ~104800000 which is roughly the default-queue-limit
What I really wanted to achieve is a queue larger than my available memory,
but I'd also like to silently dump the oldest stuff without throwing an
exception back to the producer - which leads back to my original
question......
So is it possible to set up a very large durable queue that behaves as a
ring buffer (of roughly the size of the queue as opposed to the
default-queue-limit size)
MTIA
fadams
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Is-it-possible-to-have-persistent-circular-queues-tp6158613p6166830.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to have persistent circular queues
Posted by fadams <fr...@blueyonder.co.uk>.
Many thanks, I'll give that a go.
You're a star - cheers
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Is-it-possible-to-have-persistent-circular-queues-tp6158613p6158749.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to have persistent circular queues
Posted by Gordon Sim <gs...@redhat.com>.
On 03/10/2011 05:56 PM, fadams wrote:
> Thanks that's cool.
>
> Could you please tell me how to achieve it?
>
> I think that it's the limit policy so the things are:
> "ring" and "flow-to-disk"'
>
> So I've used flow-to-disk for durable queues but I'm pretty sure that when I
> tried both it didn't work (I set it up with qpid-config).
>
> I may have screwed up the syntax - I "think" I tried --limit-policy ring
> flow-to-disk
When using qpid-config it would be:
qpid-config add queue my-durable-ring-queue --limit-policy ring
--max-queue-count N --durable
[You don't want flow-to-disk, that is a policy whereby once the limit is
reached subsequent messages are written to disk and their content s
released from memory. I personally would avoid using flow-to-disk, it
impacts performance quite significantly and doesn't actually save that
much memory unless your messages are very large.]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to have persistent circular queues
Posted by fadams <fr...@blueyonder.co.uk>.
Thanks that's cool.
Could you please tell me how to achieve it?
I think that it's the limit policy so the things are:
"ring" and "flow-to-disk"'
So I've used flow-to-disk for durable queues but I'm pretty sure that when I
tried both it didn't work (I set it up with qpid-config).
I may have screwed up the syntax - I "think" I tried --limit-policy ring
flow-to-disk
could you please tell me the right way to do it?
Thanks.
--
View this message in context: http://apache-qpid-users.2158936.n2.nabble.com/Is-it-possible-to-have-persistent-circular-queues-tp6158613p6158701.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org
Re: Is it possible to have persistent circular queues
Posted by Gordon Sim <gs...@redhat.com>.
On 03/10/2011 05:34 PM, fadams wrote:
> I know that it's possible to have in memory circular queues by setting the
> behaviour to "ring" and it's clearly possible to have persistent/durable
> queues but is it possible to make a durable queue circular?
Yes, a 'ring' queue can be durable.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:users-subscribe@qpid.apache.org