You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Toli Kuznets <tk...@marinsoftware.com> on 2013/12/31 22:52:05 UTC
ActiveMQ, Camel CBR routing issue
Have a basic question about setting up AMQ and Camel context-based routing.
We added a Camel's content based routing for ActiveMQ message systems,
but the system stops throttling when the pending messages bellow a
specific threshold ( approx ~600)
the CBR details
from("activemq:queue:Queue_In?concurrentConsumers=50&exchangePattern=InOut")
.id("UniqueRouteId")
.to("activemq:queue:Queue_Out?replyTo=Ack_Queue&concurrentConsumers=50&replyToType=Shared&requestTimeout=400000");
Another service is exposed to change the in-flight message count,
which will stop and remove the route using the unique id and re-create
the route with new value, which is working fine.
When we send ~10K messages, all messages are routed properly ( only
the specified ( in concurrentConsumers) message will be delivered to
out queue at any given time), but when the pending message is below (
appox ~600), it will send only one message at a time even though the
concurrent message count is greater than one. We tried to reduce the
memory limit (memoryUsage) in AMQ, but still if the pending message
goes below some threshold ( it reduced to ~50), it starts sending only
one message at time.
We also tried to turn off the producer control flow setting in
ActiveMQ (<policyEntry queue=">" producerFlowControl="false"
memoryLimit="1mb">), but still it stops throttling if the pending
message is bellow same threshold
We tried to add routing policy with higher max in-flight message count
( e.g:- 50000) and resume percentage of max value ( e.g:- 50), it
seems to work fine for some period of time, but again it starts
sending one message at a time later.
We are unable to find the relationship with ActiveMQ memory
parameters, producer flow controller and max in-flight message count
settings in Camel.
Has anybody seen anything similar? I wouldn't be surprised if we are
doing something stupid, but we haven't been able to find documentation
on the relationship of all of these tunable parameters.
It would be really great if you provide some ideas or documentation
for these settings.
thanks (and happy new year!)
Re: ActiveMQ, Camel CBR routing issue
Posted by Toli Kuznets <tk...@marinsoftware.com>.
Claus,
Prefetch limit may be the culprit here. We have ours set to 1 since we
have fairly long-running jobs (sometimes hours), and our volumes are
fairly low (we have bursts of 100s or 1000s every hour, and then
lulls)
What would you recommend? Raising our prefetchLimit? Seems like that
goes against the advice of setting prefetchLimit=1 for long-running
low-volume jobs, as noted in the docs (ie "If you have very few
messages and each message takes a very long time to process you might
want to set the prefetch value to 1 so that a consumer is given one
message at a time")
thanks
On Wed, Jan 1, 2014 at 2:23 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> When its something with < 1000 then it sounds like a prefetch limit
> that affects this.
>
> See more on the AMQ web site about that such as
> http://activemq.apache.org/what-is-the-prefetch-limit-for.html
>
> On Tue, Dec 31, 2013 at 10:52 PM, Toli Kuznets
> <tk...@marinsoftware.com> wrote:
>> Have a basic question about setting up AMQ and Camel context-based routing.
>> We added a Camel's content based routing for ActiveMQ message systems,
>> but the system stops throttling when the pending messages bellow a
>> specific threshold ( approx ~600)
>>
>> the CBR details
>>
>> from("activemq:queue:Queue_In?concurrentConsumers=50&exchangePattern=InOut")
>> .id("UniqueRouteId")
>> .to("activemq:queue:Queue_Out?replyTo=Ack_Queue&concurrentConsumers=50&replyToType=Shared&requestTimeout=400000");
>>
>>
>> Another service is exposed to change the in-flight message count,
>> which will stop and remove the route using the unique id and re-create
>> the route with new value, which is working fine.
>>
>> When we send ~10K messages, all messages are routed properly ( only
>> the specified ( in concurrentConsumers) message will be delivered to
>> out queue at any given time), but when the pending message is below (
>> appox ~600), it will send only one message at a time even though the
>> concurrent message count is greater than one. We tried to reduce the
>> memory limit (memoryUsage) in AMQ, but still if the pending message
>> goes below some threshold ( it reduced to ~50), it starts sending only
>> one message at time.
>>
>> We also tried to turn off the producer control flow setting in
>> ActiveMQ (<policyEntry queue=">" producerFlowControl="false"
>> memoryLimit="1mb">), but still it stops throttling if the pending
>> message is bellow same threshold
>>
>> We tried to add routing policy with higher max in-flight message count
>> ( e.g:- 50000) and resume percentage of max value ( e.g:- 50), it
>> seems to work fine for some period of time, but again it starts
>> sending one message at a time later.
>>
>> We are unable to find the relationship with ActiveMQ memory
>> parameters, producer flow controller and max in-flight message count
>> settings in Camel.
>>
>> Has anybody seen anything similar? I wouldn't be surprised if we are
>> doing something stupid, but we haven't been able to find documentation
>> on the relationship of all of these tunable parameters.
>>
>> It would be really great if you provide some ideas or documentation
>> for these settings.
>>
>> thanks (and happy new year!)
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io
Re: ActiveMQ, Camel CBR routing issue
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
When its something with < 1000 then it sounds like a prefetch limit
that affects this.
See more on the AMQ web site about that such as
http://activemq.apache.org/what-is-the-prefetch-limit-for.html
On Tue, Dec 31, 2013 at 10:52 PM, Toli Kuznets
<tk...@marinsoftware.com> wrote:
> Have a basic question about setting up AMQ and Camel context-based routing.
> We added a Camel's content based routing for ActiveMQ message systems,
> but the system stops throttling when the pending messages bellow a
> specific threshold ( approx ~600)
>
> the CBR details
>
> from("activemq:queue:Queue_In?concurrentConsumers=50&exchangePattern=InOut")
> .id("UniqueRouteId")
> .to("activemq:queue:Queue_Out?replyTo=Ack_Queue&concurrentConsumers=50&replyToType=Shared&requestTimeout=400000");
>
>
> Another service is exposed to change the in-flight message count,
> which will stop and remove the route using the unique id and re-create
> the route with new value, which is working fine.
>
> When we send ~10K messages, all messages are routed properly ( only
> the specified ( in concurrentConsumers) message will be delivered to
> out queue at any given time), but when the pending message is below (
> appox ~600), it will send only one message at a time even though the
> concurrent message count is greater than one. We tried to reduce the
> memory limit (memoryUsage) in AMQ, but still if the pending message
> goes below some threshold ( it reduced to ~50), it starts sending only
> one message at time.
>
> We also tried to turn off the producer control flow setting in
> ActiveMQ (<policyEntry queue=">" producerFlowControl="false"
> memoryLimit="1mb">), but still it stops throttling if the pending
> message is bellow same threshold
>
> We tried to add routing policy with higher max in-flight message count
> ( e.g:- 50000) and resume percentage of max value ( e.g:- 50), it
> seems to work fine for some period of time, but again it starts
> sending one message at a time later.
>
> We are unable to find the relationship with ActiveMQ memory
> parameters, producer flow controller and max in-flight message count
> settings in Camel.
>
> Has anybody seen anything similar? I wouldn't be surprised if we are
> doing something stupid, but we haven't been able to find documentation
> on the relationship of all of these tunable parameters.
>
> It would be really great if you provide some ideas or documentation
> for these settings.
>
> thanks (and happy new year!)
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io