Broker with different queue behaviors when full

[Mike C <sm...@gmail.com>]

Hi,

I have a broker that has 2 queues: important.queue & problems.queue,

The important.queue contains messages that I need to process, and if I
have problems processing any of the messages I put the message into
the problems.queue with a bit of meta-data so I can look at it later.
It is like a DLQ but not just for unreachable destinations, and also
contains any meta-data gleaned from the consumer as well as stack
traces etc. So I can't just use the existing DLQ infrastructure of
ActiveMQ.

Is there any way to specify different storage characteristics for each
queue? I want the important.queue to block upon out of space, but the
problems.queue I don't care about soo much and would prefer to
FIFO-style evict the oldest messages from the queue to make space for
new messages. The problems.queue is persistent as while I don't care
about it as much as the incoming data, it still does have some valid
uses so should do its best to survive a full crash (power outage,
vm/OS crash, etc...).

Is there a way to do this using ActiveMQ configuration? Essentially
I'm trying to get a blocking queue and a non-blocking silent-purge
queue, each with different usage limits, in a single broker.

Pretty much all the techniques out there are only for topics. I did
see 'sendFailIfNoSpace' however that applies to overall systemUsage. I
want to ensure that the problems.queue only takes up a fraction of the
system resource. I've tried specifying destinationPolicy/policyEntry
memoryLimit to give the problems queue less memory than the important
queue and it doesn't seem to make a difference (presumably the data is
just being written straight to disk rather than stored in-memory?).
Ideas?

Thanks,

Mike

Re: Broker with different queue behaviors when full

[Gary Tully <ga...@gmail.com>]

A first step would be to use a time to live on the error queue, so that old
messages get deleted. This is a message/producer property, but you could
build a plugin to apply a per destination expiry value  broker side, using
http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/main/java/org/apache/activemq/plugin/ForcePersistencyModeBrokerPlugin.java?revision=812514&view=markup&pathrev=812514as
inspiration

To configure different store usage limits there is a per destination
storeUsageHighWaterMark, so you could configure your important queue to
block or throw the sendFailIfNoSpace at 50% usage leaving the other 50% for
the error queue.


w.r.t to memory usage. the default cursor will simply not cache messages if
there is no space, so you won't get blocking behavior.

To restrict memory usage and get blocking, configure the vmQueueCursor for
the important queue - message dispatch will be from memory and you can get
flow control to kick in using per destination memory limits.

The error queue will simply stop caching messages with the default store
cursor.



On 10 December 2012 15:51, Mike C <sm...@gmail.com> wrote:

> Hi,
>
> I have a broker that has 2 queues: important.queue & problems.queue,
>
> The important.queue contains messages that I need to process, and if I
> have problems processing any of the messages I put the message into
> the problems.queue with a bit of meta-data so I can look at it later.
> It is like a DLQ but not just for unreachable destinations, and also
> contains any meta-data gleaned from the consumer as well as stack
> traces etc. So I can't just use the existing DLQ infrastructure of
> ActiveMQ.
>
> Is there any way to specify different storage characteristics for each
> queue? I want the important.queue to block upon out of space, but the
> problems.queue I don't care about soo much and would prefer to
> FIFO-style evict the oldest messages from the queue to make space for
> new messages. The problems.queue is persistent as while I don't care
> about it as much as the incoming data, it still does have some valid
> uses so should do its best to survive a full crash (power outage,
> vm/OS crash, etc...).
>
> Is there a way to do this using ActiveMQ configuration? Essentially
> I'm trying to get a blocking queue and a non-blocking silent-purge
> queue, each with different usage limits, in a single broker.
>
> Pretty much all the techniques out there are only for topics. I did
> see 'sendFailIfNoSpace' however that applies to overall systemUsage. I
> want to ensure that the problems.queue only takes up a fraction of the
> system resource. I've tried specifying destinationPolicy/policyEntry
> memoryLimit to give the problems queue less memory than the important
> queue and it doesn't seem to make a difference (presumably the data is
> just being written straight to disk rather than stored in-memory?).
> Ideas?
>
> Thanks,
>
> Mike
>



-- 
http://redhat.com
http://blog.garytully.com