Limit number of JMS messages when processing asynchronously

[jpeschke <pe...@etone.de>]

Hello all,
We have a system consuming messages at a high rate from a JMS queue
asynchronously and parallel. For Performance reasons, we have a very high
idle timeout, so normally our camel-based software never reconnects to the
broker and stays connected.

The problem is that it seems that the asynchronous behaviour lets Camel take
as many JMS messages as it can get, so sometimes we have thousands of
inflight messages waiting to be processed. This is not such a big problem
during normal operation, but when we shut down the system for maintenance,
sometimes there are so many messages that they cannot be processed during a
graceful shutdown.

My question is: How can I specify how many messages camel takes out of the
queue when using an asynchonrous consumer?

I already defined
- idleConsumerLimit
- concurrentConsumers
- maxConcurrentConsumers
- maxMessagesPerTask

but that doesn't seem to work (as far as I understand because these settings
influence how many threads are used to get messages out of the queue but not
how many messages may be "pumped" into the camel route)

Any ides?

Thank you & best regards,
Joerg






--
View this message in context: http://camel.465427.n5.nabble.com/Limit-number-of-JMS-messages-when-processing-asynchronously-tp5774822.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Limit number of JMS messages when processing asynchronously

[jpeschke <pe...@etone.de>]

Thank you four your ideas, Claus.
We need the "asyncConsumer=true" setting as it dramatically boosts our
performance (the broker already delivers the messages unsorted, so we have
to re-order them anyway). 

But a route policy should do the trick - I have already thought about
implementing a RoutePolicy to turn off the JMS consumer in some cases (The
processed JMS messages are buffered in a database, another camel route is
used to fetch & aggregate them. If there are many messages in the JMS queue,
we sometimes experience the problem that database reads are blocked by the
write operations of the JMS consumer until the traffic decreases - so for
periods of high traffic, it could be useful to disable the JMS consumer from
time to time to give the database consumer the chance to kick in)

So - RoutePolicy, here we go!


Thank you,
Joerg




--
View this message in context: http://camel.465427.n5.nabble.com/Limit-number-of-JMS-messages-when-processing-asynchronously-tp5774822p5774828.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Limit number of JMS messages when processing asynchronously

[Claus Ibsen <cl...@gmail.com>]

It depends a bit what you do with these messages in the Camel routes.

The jms consumer is in fact sequential by default (so if the routing
at some point becomes async, then it will wait for that inflight
message to complete before it consumes the next). But you can turn
that off with asyncConsumer=true, and if so the consumer will run as
fast as it can.

Using seda queues or other async components in the route all can have
an influence how it all flows.

Many of those jms settings is for tuning how the jms consumer react on
load, so it can scale up and down dending on how many of threads are
processing or idle, vs if there is no messages on the queue vs many
messages etc.

You can also use the throttler route policy
http://camel.apache.org/routepolicy

that can suspend/resume the routes depending on number of inflight messages.

On Tue, Dec 8, 2015 at 1:57 PM, jpeschke <pe...@etone.de> wrote:
> Hello all,
> We have a system consuming messages at a high rate from a JMS queue
> asynchronously and parallel. For Performance reasons, we have a very high
> idle timeout, so normally our camel-based software never reconnects to the
> broker and stays connected.
>
> The problem is that it seems that the asynchronous behaviour lets Camel take
> as many JMS messages as it can get, so sometimes we have thousands of
> inflight messages waiting to be processed. This is not such a big problem
> during normal operation, but when we shut down the system for maintenance,
> sometimes there are so many messages that they cannot be processed during a
> graceful shutdown.
>
> My question is: How can I specify how many messages camel takes out of the
> queue when using an asynchonrous consumer?
>
> I already defined
> - idleConsumerLimit
> - concurrentConsumers
> - maxConcurrentConsumers
> - maxMessagesPerTask
>
> but that doesn't seem to work (as far as I understand because these settings
> influence how many threads are used to get messages out of the queue but not
> how many messages may be "pumped" into the camel route)
>
> Any ides?
>
> Thank you & best regards,
> Joerg
>
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Limit-number-of-JMS-messages-when-processing-asynchronously-tp5774822.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2