Message throttling problem

[krishananth75 <kr...@gmail.com>]

Hi,

We would like to throttle the number of messages at any point of time to 20
from Active MQ.
Few other constraints are:
a) Total number of messages received by all consumers should be maximum of
20
b) Some of them might be slow consumers.  Let us say 5 of them are still
processing,
then MQ should send only 20 - 5 = 15 messages to remaining consumers

Initial approach:
We would like producers to post messages to a (distributed) Topic, use Camel
framework 
to forward the message to corresponding Active MQ Queue based on a Header
parameter.

Problem:
a) The Camel throttler by default, on second zero, throttles 20 messages
from Topic
and all of them reach their Queues for Consumers to use  
b) At second 1, the Camel throttler forwards another 20 messages from Topic 
and all of them reach their Queues by doubling the messages in the Queue

Is there a way (without setting delay) in Camel Throttler to dispatch only N
number of messages based on destination Queue count?

Note:
I had the queuePrefetch (to true), producerFlowControl (to true) in Active
MQ XML for all the Queues.
But none of the Queues are blocking new messages; they accept the messages
forwarded by Camel plugin.

Are there any parameters available in Active MQ or Camel to restrict message
limits based on consumption?

thank you,

Ananth Krishnan



--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-throttling-problem-tp4684744.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Message throttling problem

[artnaseef <ar...@artnaseef.com>]

That's an interesting use-case.  I suspect there are ways to accomplish this
within ActiveMQ, but I wouldn't recommend going that route - you'll end up
in areas of the code that get less attention and solving problems into which
you run will be harder since it doesn't match the use-cases on which
ActiveMQ developers are frequently working.

ActiveMQ is focused on asynchronous messaging, which is push-based and
requires (as stated in the JMS spec) continuous, timely receipt of messages. 
It is not a store nor a tool for queries (e.g. "get me the last 20").

Something like EHCache or hazelcast may work better for the use case - based
on my limited understanding of the problem.

For example, if I wanted to always have the last 20 items, and only the last
20, of a stream of items available to query, the following comes to mind:

* Producer into an ActiveMQ queue
* Consumer from ActiveMQ updates EHCache with each new message
** EHCache structure maintains a limited-size queue
** New entries pushed into the queue
** Once size limit reached, drop the oldest from the queue
** Use EHCache locking to guarantee the queue size is atomically updated




--
View this message in context: http://activemq.2283324.n4.nabble.com/Message-throttling-problem-tp4684744p4684748.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.