You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@qpid.apache.org by jbelch <ja...@verizon.net> on 2013/08/05 17:55:56 UTC
Automatic queue cleanup
In our test environment, we bring applications up and down quite a bit.
Sometimes we will leave applications that are producing Qpid messages up for
an extended period of time and we may or may not have the Qpic client
applications up a running. When we do this, we notice that the Java Broker
will run out of memory. Is there a configuration property that you can put
in virtualhosts.xml so the the Java Broker will reclaim memory when a queue
reaches a certain size? Is there a preferred method to prevent the Java
Qpid broker from running out of memory?
--
View this message in context: http://qpid.2158936.n2.nabble.com/Automatic-queue-cleanup-tp7596467.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Automatic queue cleanup
Posted by Robbie Gemmell <ro...@gmail.com>.
Hi James,
Rob meant they would be evicted from memory if required (see [1] for a mail
that covers some background on a semi-related topic), the messages will
still be in the persistent store until they are removed from the queue.
The Java broker currently supports the following ways of removing messages
from queues:
1) Consuming them.
2) Expiration due to TTL.
3) Deleting/moving them using the management interfaces.
4) The queue itself being deleted.
If the messages being sent are persistent, the data/headers for these can
exceed theJVM heap size, but the broker requires heap memory to service
connections, to store non-persistent messages in their entirety, and to
store all the objects for queues etc and the messages on them (whether
persistent or not). You will need to size the JVM heap according to your
requirements, considering elemetns such as how many connections there will
be, how many messages you might expect to be backlogged, are there
non-persistent messages, etc etc. Using a 3-6GB heap can be fairly typical,
but what you need will depend on your use case. Out of interest, do you
have an idea of how many (and what type, size etc of) messages were on the
broker when it ran out of memory?
Robbie
[1]http://mail-archives.apache.org/mod_mbox/qpid-users/201304.mbox/%3CCAFitrpR906nHnYb1RQnYTvVzpkChzEohYF6iGDQJ0rj4-bOYFQ%40mail.gmail.com%3E
On 5 August 2013 22:57, jbelch <ja...@verizon.net> wrote:
> Rob,
>
> When you say the Broker will evict the messages, what exactly happens
> behind the scenes? Does it evict the messages from the heap and also
> delete
> from the Berkeley DB store? Do it evict the messages from the heap and
> retain the message in the Berkeley DB store so that the peristent messages
> can be reloaded when the Broker is cycle. We would prefer that the
> messages
> get deleted from the Berkeley DB store and they are deleted from memory.
>
> Thanks,
> James
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Automatic-queue-cleanup-tp7596467p7596495.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Automatic queue cleanup
Posted by jbelch <ja...@verizon.net>.
Rob,
When you say the Broker will evict the messages, what exactly happens
behind the scenes? Does it evict the messages from the heap and also delete
from the Berkeley DB store? Do it evict the messages from the heap and
retain the message in the Berkeley DB store so that the peristent messages
can be reloaded when the Broker is cycle. We would prefer that the messages
get deleted from the Berkeley DB store and they are deleted from memory.
Thanks,
James
--
View this message in context: http://qpid.2158936.n2.nabble.com/Automatic-queue-cleanup-tp7596467p7596495.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Automatic queue cleanup
Posted by jbelch <ja...@verizon.net>.
We do persist some queues using Berkeley DB, but we don't want to persist
everything, so it sounds like we may need to start setting ttl on all of our
messages. We currently have ttl set on most of the messages but not all the
messages. Once we have it set on everything, it sounds like that should
help matters. I think we would definitely be interested in the patch that
Alex produced for implementing ring queues in the Java Broker. What version
of Qpid was this patch created for and will it be available in a future
release?
--
View this message in context: http://qpid.2158936.n2.nabble.com/Automatic-queue-cleanup-tp7596467p7596492.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Automatic queue cleanup
Posted by Rob Godfrey <ro...@gmail.com>.
Hi James,
if you are using persistent messages sent to durable queues then the Java
Broker should be able to evict the message content from memory when low
memory conditions are reached (though it does retain some overhead for meta
data relating to the entry of the message on the queue). Messages not
marked as persistent will retain all their content in memory (and thus if
not restricted by other means, such as TTL, or producer flow control),
excessive queue size may cause out of memory exceptions.
In terms of controlling the amount of message data retained by the broker
in memory, there are currently a number of steps you could take... firstly
- as above - ensuring that you are using persistent messages. Secondly you
can use TTL to cause message expiry (if this is appropriate for your use
case). You can also use producer flow control to enforce (approximate)
limits on how large queues can grow.
Unfortunately the Java Broker does not yet implement all the queue policies
that the C++ Broker implements... in particular there is no way of limiting
the size or depth of a queue, and discarding messages in either a FIFO or
LIFO manner - is this the sort of functionality you would be looking for?
I know Alex produced a patch at some point for implementing ring queues in
the Java Broker, but we've not yet seen a great deal of demand for this
behaviour... however I believe Fraser outlined a use case where it might be
useful, and possibly you also have such a use case
-- Rob
On 5 August 2013 21:40, jbelch <ja...@verizon.net> wrote:
> Keith,
>
> We have increased it to 2GB in the qpid-server. We could continue to
> increase it, but we wanted to figure out if there was a way to get the Qpid
> broker to cleanup. We are using Berkeley DB as the message store and we
> only have two queues marked as durable right now. The errors I see right
> before it crashes are:
>
> MESSAGE_COUNT_ALERT on Queue <queue> - 180: Maximum count on queue
> threshold
> (50) breached
> MESSAGE_AGE_ALERT on Queue <queue> - 343925s: Maximum age on queue
> threshold
> (3600s) breached
>
>
>
> --
> View this message in context:
> http://qpid.2158936.n2.nabble.com/Automatic-queue-cleanup-tp7596467p7596486.html
> Sent from the Apache Qpid users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>
>
Re: Automatic queue cleanup
Posted by jbelch <ja...@verizon.net>.
Keith,
We have increased it to 2GB in the qpid-server. We could continue to
increase it, but we wanted to figure out if there was a way to get the Qpid
broker to cleanup. We are using Berkeley DB as the message store and we
only have two queues marked as durable right now. The errors I see right
before it crashes are:
MESSAGE_COUNT_ALERT on Queue <queue> - 180: Maximum count on queue threshold
(50) breached
MESSAGE_AGE_ALERT on Queue <queue> - 343925s: Maximum age on queue threshold
(3600s) breached
--
View this message in context: http://qpid.2158936.n2.nabble.com/Automatic-queue-cleanup-tp7596467p7596486.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org
Re: Automatic queue cleanup
Posted by Keith W <ke...@gmail.com>.
Hi James
You can control the amount of heap memory allocated to the Broker by
setting the QPID_JAVA_MEM environment option before you call qpid-server
script. It accepts standard JVM memory options e.g. -Xmx4g.
However, I'm curious to understand more about your failure case. What
store are you using (bdb, derby)? Are the queues marked durable? Do you
have the stack trace?
It would be informative if you can reproduce the issue with the broker
logging turned up to debug (change the org.apache.qpid logger within
etc/log4j.xml (line 92) from warn to debug). I am especially interested in
the debug produced by the class VirtualHostConfigRecoveryHandler. (Either
post output to this thread (please ellipse repeative lines or use pastebin).
Kind regards, Keith.