You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Claus Ibsen <cl...@gmail.com> on 2011/08/09 14:25:27 UTC
[DISCUSS] - JmsConsumer support for asynchronous routing engine -
Higher scalability
Hi
See ticket
https://issues.apache.org/jira/browse/CAMEL-3632
Take a look at the following routes
from("activemq:queue:inbox?synchronous=true")
.to("jetty:http://0.0.0.0:9432/myapp")
.to("log:result?groupSize=10", "mock:result");
from("jetty:http://0.0.0.0:9432/myapp")
.delay(100)
.transform(body().prepend("Bye "));
Its a fairly simple flow JMS -> JETTY -> MOCK
The JMS consumer will by default be in synchronous mode. Ticket
CAMEL-3632 is to optimize this by supporting asynchronous routing if
the JMS consumer is *not* transactional, and *not* sending back a
reply message.
So the scenarios is for high performance one way messaging.
The 2nd route above simulate a little processing time, which takes 100 millis.
If we run a test where we send 1000 messages to the queue. How long
time would it take to process those messages synchronously? Of course
roughly 1000 * 0.1 sec = 100 sec. On my laptop it takes:
Took: 1 minute (105393 millis)
Now imagine we set synchronous=false on the JMS consumer so it runs
asynchronously. How fast would it then go? Well lets try
Took: 10.742 seconds (10742 millis)
That is a lot fast, well of couse its about 10x faster, since the JMS
consumer is not blocked waiting for the JETTY reply.
However the messages is now not processed in sequence, that is the cost.
To note in both scenarios this was done with 1 JMS consumer thread. I
did not enable concurrentConsumers. So with the async routing engine
we could scale and utilize the CPU resources better.
This seems like a great addition. And if we add this, we should add
some documentation with the pros/cons for using this.
My question is what would a sensitive default setting be? Should we
let the JMS consumer be kept as synchronous by default, to keep it
fully backwards compatible. Then end users must manually set
synchronous=false on the JMS endpoint to enable that.
Any thoughts?
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: [DISCUSS] - JmsConsumer support for asynchronous routing engine -
Higher scalability
Posted by Claus Ibsen <cl...@gmail.com>.
On Wed, Aug 10, 2011 at 11:12 AM, Charles Moulliard
<cm...@gmail.com> wrote:
> Hi Claus,
>
> I would prefer that we follow the same convention for JMS, NMR and all
> related components. As they are by default async (This is why we use a
> MOM or NMR bus), the producer and consumer should be also defined as
> asynchronous by default and changed by the user when they would like
> to switch to synchronous mode.
>
A good idea. Having slept on this I also lean towards this, to keep it
consistent with the other components that supports asynchronous
routing engine.
http://camel.apache.org/asynchronous-routing-engine.html
> Just 2 cents to the debate.
>
> Regards,
>
> Charles
>
>
> On Tue, Aug 9, 2011 at 2:25 PM, Claus Ibsen <cl...@gmail.com> wrote:
>> Hi
>>
>> See ticket
>> https://issues.apache.org/jira/browse/CAMEL-3632
>>
>> Take a look at the following routes
>>
>> from("activemq:queue:inbox?synchronous=true")
>> .to("jetty:http://0.0.0.0:9432/myapp")
>> .to("log:result?groupSize=10", "mock:result");
>>
>> from("jetty:http://0.0.0.0:9432/myapp")
>> .delay(100)
>> .transform(body().prepend("Bye "));
>>
>>
>> Its a fairly simple flow JMS -> JETTY -> MOCK
>>
>> The JMS consumer will by default be in synchronous mode. Ticket
>> CAMEL-3632 is to optimize this by supporting asynchronous routing if
>> the JMS consumer is *not* transactional, and *not* sending back a
>> reply message.
>>
>> So the scenarios is for high performance one way messaging.
>>
>> The 2nd route above simulate a little processing time, which takes 100 millis.
>>
>> If we run a test where we send 1000 messages to the queue. How long
>> time would it take to process those messages synchronously? Of course
>> roughly 1000 * 0.1 sec = 100 sec. On my laptop it takes:
>> Took: 1 minute (105393 millis)
>>
>> Now imagine we set synchronous=false on the JMS consumer so it runs
>> asynchronously. How fast would it then go? Well lets try
>> Took: 10.742 seconds (10742 millis)
>>
>> That is a lot fast, well of couse its about 10x faster, since the JMS
>> consumer is not blocked waiting for the JETTY reply.
>> However the messages is now not processed in sequence, that is the cost.
>>
>> To note in both scenarios this was done with 1 JMS consumer thread. I
>> did not enable concurrentConsumers. So with the async routing engine
>> we could scale and utilize the CPU resources better.
>>
>> This seems like a great addition. And if we add this, we should add
>> some documentation with the pros/cons for using this.
>>
>> My question is what would a sensitive default setting be? Should we
>> let the JMS consumer be kept as synchronous by default, to keep it
>> fully backwards compatible. Then end users must manually set
>> synchronous=false on the JMS endpoint to enable that.
>>
>> Any thoughts?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: [DISCUSS] - JmsConsumer support for asynchronous routing engine -
Higher scalability
Posted by Charles Moulliard <cm...@gmail.com>.
Hi Claus,
I would prefer that we follow the same convention for JMS, NMR and all
related components. As they are by default async (This is why we use a
MOM or NMR bus), the producer and consumer should be also defined as
asynchronous by default and changed by the user when they would like
to switch to synchronous mode.
Just 2 cents to the debate.
Regards,
Charles
On Tue, Aug 9, 2011 at 2:25 PM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> See ticket
> https://issues.apache.org/jira/browse/CAMEL-3632
>
> Take a look at the following routes
>
> from("activemq:queue:inbox?synchronous=true")
> .to("jetty:http://0.0.0.0:9432/myapp")
> .to("log:result?groupSize=10", "mock:result");
>
> from("jetty:http://0.0.0.0:9432/myapp")
> .delay(100)
> .transform(body().prepend("Bye "));
>
>
> Its a fairly simple flow JMS -> JETTY -> MOCK
>
> The JMS consumer will by default be in synchronous mode. Ticket
> CAMEL-3632 is to optimize this by supporting asynchronous routing if
> the JMS consumer is *not* transactional, and *not* sending back a
> reply message.
>
> So the scenarios is for high performance one way messaging.
>
> The 2nd route above simulate a little processing time, which takes 100 millis.
>
> If we run a test where we send 1000 messages to the queue. How long
> time would it take to process those messages synchronously? Of course
> roughly 1000 * 0.1 sec = 100 sec. On my laptop it takes:
> Took: 1 minute (105393 millis)
>
> Now imagine we set synchronous=false on the JMS consumer so it runs
> asynchronously. How fast would it then go? Well lets try
> Took: 10.742 seconds (10742 millis)
>
> That is a lot fast, well of couse its about 10x faster, since the JMS
> consumer is not blocked waiting for the JETTY reply.
> However the messages is now not processed in sequence, that is the cost.
>
> To note in both scenarios this was done with 1 JMS consumer thread. I
> did not enable concurrentConsumers. So with the async routing engine
> we could scale and utilize the CPU resources better.
>
> This seems like a great addition. And if we add this, we should add
> some documentation with the pros/cons for using this.
>
> My question is what would a sensitive default setting be? Should we
> let the JMS consumer be kept as synchronous by default, to keep it
> fully backwards compatible. Then end users must manually set
> synchronous=false on the JMS endpoint to enable that.
>
> Any thoughts?
>
>
>
>
>
>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: [DISCUSS] - JmsConsumer support for asynchronous routing engine
- Higher scalability
Posted by tmi <mi...@web.de>.
With regards to performance improvements when using async engine, I am
wondering if you get comparable performance by using concurrentConsumers on
the camel-jms endpoint.
However I can see the benefit of not using multiple threads for fast
routing.
--
View this message in context: http://camel.465427.n5.nabble.com/DISCUSS-JmsConsumer-support-for-asynchronous-routing-engine-Higher-scalability-tp4681859p4758522.html
Sent from the Camel Development mailing list archive at Nabble.com.
Re: [DISCUSS] - JmsConsumer support for asynchronous routing engine -
Higher scalability
Posted by Charles Moulliard <cm...@gmail.com>.
Hi Christian,
We need to keep async/sync properties at the level of the JMS
component itself as this behavior belongs to the way we can use a MOM
or ActiveMQ. Nevertheless, in some case, this behavior could be
changed (but that means that documentation should be clear on that)
that async becomes sync if we have other pieces of the camel route
designed as sync. But be careful to avoid to introduce too much
confusion in the head of the users due to lack of claritity and
information.
Regards,
Charles
On Tue, Aug 9, 2011 at 6:20 PM, Christian Schneider
<ch...@die-schneider.net> wrote:
> Hi Claus,
>
> without changing anything in camel wouldn´t a simple change in the route
> also work?
>
> from("activemq:queue:inbox?synchronous=true")
> .threads()
> .to("jetty:http://0.0.0.0:9432/myapp")
> .to("log:result?groupSize=10", "mock:result");
>
> So I ask myself if we really need async support in the jms consumer if we
> can support the scenario using a well known dsl element.
>
> Christian
>
>
> Am 09.08.2011 14:25, schrieb Claus Ibsen:
>>
>> Hi
>>
>> See ticket
>> https://issues.apache.org/jira/browse/CAMEL-3632
>>
>> Take a look at the following routes
>>
>> from("activemq:queue:inbox?synchronous=true")
>> .to("jetty:http://0.0.0.0:9432/myapp")
>> .to("log:result?groupSize=10", "mock:result");
>>
>> from("jetty:http://0.0.0.0:9432/myapp")
>> .delay(100)
>> .transform(body().prepend("Bye "));
>>
>>
>> Its a fairly simple flow JMS -> JETTY -> MOCK
>>
>> The JMS consumer will by default be in synchronous mode. Ticket
>> CAMEL-3632 is to optimize this by supporting asynchronous routing if
>> the JMS consumer is *not* transactional, and *not* sending back a
>> reply message.
>>
>> So the scenarios is for high performance one way messaging.
>>
>> The 2nd route above simulate a little processing time, which takes 100
>> millis.
>>
>> If we run a test where we send 1000 messages to the queue. How long
>> time would it take to process those messages synchronously? Of course
>> roughly 1000 * 0.1 sec = 100 sec. On my laptop it takes:
>> Took: 1 minute (105393 millis)
>>
>> Now imagine we set synchronous=false on the JMS consumer so it runs
>> asynchronously. How fast would it then go? Well lets try
>> Took: 10.742 seconds (10742 millis)
>>
>> That is a lot fast, well of couse its about 10x faster, since the JMS
>> consumer is not blocked waiting for the JETTY reply.
>> However the messages is now not processed in sequence, that is the cost.
>>
>> To note in both scenarios this was done with 1 JMS consumer thread. I
>> did not enable concurrentConsumers. So with the async routing engine
>> we could scale and utilize the CPU resources better.
>>
>> This seems like a great addition. And if we add this, we should add
>> some documentation with the pros/cons for using this.
>>
>> My question is what would a sensitive default setting be? Should we
>> let the JMS consumer be kept as synchronous by default, to keep it
>> fully backwards compatible. Then end users must manually set
>> synchronous=false on the JMS endpoint to enable that.
>>
>> Any thoughts?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>
Re: [DISCUSS] - JmsConsumer support for asynchronous routing engine -
Higher scalability
Posted by Marat Bedretdinov <ma...@gmail.com>.
I think one more important downside point of using user threads to dispatch
messages from a JMS provider is that you loose guaranteed order of delivery
and the onus of sorting them and guaranteeing message delivery to recipients
is now put onto application code.
I think you would only want to do that in a case where message loss does not
break the application logic and message throughput is more important than
getting all of the produced messages in order. But once you go there why not
use non durable Topics? This is exactly the use case they were designed for.
For example a News Feed application. Each user thread would receive a
message on the topic and deliver it to the destination in parallel.
On Wed, Aug 10, 2011 at 5:48 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Tue, Aug 9, 2011 at 6:20 PM, Christian Schneider
> <ch...@die-schneider.net> wrote:
> > Hi Claus,
> >
> > without changing anything in camel wouldn´t a simple change in the route
> > also work?
> >
> > from("activemq:queue:inbox?synchronous=true")
> > .threads()
> > .to("jetty:http://0.0.0.0:9432/myapp")
> > .to("log:result?groupSize=10", "mock:result");
> >
> > So I ask myself if we really need async support in the jms consumer if we
> > can support the scenario using a well known dsl element.
> >
>
> Using threads is not exactly the same. But its is related.
>
> Without threads, you only have the thread of the JMS consumer. And the
> JMS consumer thread will run at a pace that the system can handle. It
> will now consume messages from the JMS broker, faster than Camel can
> handle it. For example if the other pieces in the routes are
> synchronous, then it will keep being synchronous. Its only because
> Jetty supports asynchronous, we are able to scale so high using JMS +
> Jetty.
>
> Here is a link with the components that native support async routing engine
> http://camel.apache.org/asynchronous-routing-engine.html
>
> If you use threads DSL, then you introduce a thread pool, with task
> queue layer. So now you have more active threads in the system. The
> threads DSL have a task queue, so the JMS consumer would be able to
> run faster, then Camel can handle it. By this I mean the JMS consumer
> can consume JMS messages from the JMS broker so fast, and hand over
> the messages to the threads DSL, and have the message stored in the
> thread pool task queue. And it can do this faster than the worker
> threads from the thread pool, so messages will pile up. To remedy this
> you would have to throttle the JMS consumer, or add more worker
> threads, if that is possible. Also you have the complexity what to do
> if the task queue has an upper limit, should you reject the task,
> wait, discard existing tasks, or steal the consumer thread, and have
> it execute as it was a worker thread etc. Also the JMS message would
> be acknowledged and thus consumed from the JMS broker point of view.
> But they are stored in memory in the threads DSL task queue. So in
> case of a crash, or a shutdown, you would have to wait for all those
> tasks to complete. All together you have a more complicated situation.
>
> You can of course customize the threads DSL to have a lower task
> queue. In fact you would be able to have no task queue at all (size of
> 0) but then you have the overhead of handing over the message from the
> JMS consumer to a thread from the threads DSL etc.
>
> Regardless what the current logic in the JmsConsumer is fully
> synchronous so even if you use threads, the JmsConsumer would block.
> So we would have to improve the source code to support this.
>
> And in the future, eg possible with Camel 3.0+ we will support
> asynchronous transactions as well, then this becomes even more
> interesting.
>
>
>
> > Christian
> >
> >
> > Am 09.08.2011 14:25, schrieb Claus Ibsen:
> >>
> >> Hi
> >>
> >> See ticket
> >> https://issues.apache.org/jira/browse/CAMEL-3632
> >>
> >> Take a look at the following routes
> >>
> >> from("activemq:queue:inbox?synchronous=true")
> >> .to("jetty:http://0.0.0.0:9432/myapp")
> >> .to("log:result?groupSize=10", "mock:result");
> >>
> >> from("jetty:http://0.0.0.0:9432/myapp")
> >> .delay(100)
> >> .transform(body().prepend("Bye "));
> >>
> >>
> >> Its a fairly simple flow JMS -> JETTY -> MOCK
> >>
> >> The JMS consumer will by default be in synchronous mode. Ticket
> >> CAMEL-3632 is to optimize this by supporting asynchronous routing if
> >> the JMS consumer is *not* transactional, and *not* sending back a
> >> reply message.
> >>
> >> So the scenarios is for high performance one way messaging.
> >>
> >> The 2nd route above simulate a little processing time, which takes 100
> >> millis.
> >>
> >> If we run a test where we send 1000 messages to the queue. How long
> >> time would it take to process those messages synchronously? Of course
> >> roughly 1000 * 0.1 sec = 100 sec. On my laptop it takes:
> >> Took: 1 minute (105393 millis)
> >>
> >> Now imagine we set synchronous=false on the JMS consumer so it runs
> >> asynchronously. How fast would it then go? Well lets try
> >> Took: 10.742 seconds (10742 millis)
> >>
> >> That is a lot fast, well of couse its about 10x faster, since the JMS
> >> consumer is not blocked waiting for the JETTY reply.
> >> However the messages is now not processed in sequence, that is the cost.
> >>
> >> To note in both scenarios this was done with 1 JMS consumer thread. I
> >> did not enable concurrentConsumers. So with the async routing engine
> >> we could scale and utilize the CPU resources better.
> >>
> >> This seems like a great addition. And if we add this, we should add
> >> some documentation with the pros/cons for using this.
> >>
> >> My question is what would a sensitive default setting be? Should we
> >> let the JMS consumer be kept as synchronous by default, to keep it
> >> fully backwards compatible. Then end users must manually set
> >> synchronous=false on the JMS endpoint to enable that.
> >>
> >> Any thoughts?
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> > --
> > Christian Schneider
> > http://www.liquid-reality.de
> >
> > Open Source Architect
> > http://www.talend.com
> >
> >
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: [DISCUSS] - JmsConsumer support for asynchronous routing engine -
Higher scalability
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Aug 9, 2011 at 6:20 PM, Christian Schneider
<ch...@die-schneider.net> wrote:
> Hi Claus,
>
> without changing anything in camel wouldn´t a simple change in the route
> also work?
>
> from("activemq:queue:inbox?synchronous=true")
> .threads()
> .to("jetty:http://0.0.0.0:9432/myapp")
> .to("log:result?groupSize=10", "mock:result");
>
> So I ask myself if we really need async support in the jms consumer if we
> can support the scenario using a well known dsl element.
>
Using threads is not exactly the same. But its is related.
Without threads, you only have the thread of the JMS consumer. And the
JMS consumer thread will run at a pace that the system can handle. It
will now consume messages from the JMS broker, faster than Camel can
handle it. For example if the other pieces in the routes are
synchronous, then it will keep being synchronous. Its only because
Jetty supports asynchronous, we are able to scale so high using JMS +
Jetty.
Here is a link with the components that native support async routing engine
http://camel.apache.org/asynchronous-routing-engine.html
If you use threads DSL, then you introduce a thread pool, with task
queue layer. So now you have more active threads in the system. The
threads DSL have a task queue, so the JMS consumer would be able to
run faster, then Camel can handle it. By this I mean the JMS consumer
can consume JMS messages from the JMS broker so fast, and hand over
the messages to the threads DSL, and have the message stored in the
thread pool task queue. And it can do this faster than the worker
threads from the thread pool, so messages will pile up. To remedy this
you would have to throttle the JMS consumer, or add more worker
threads, if that is possible. Also you have the complexity what to do
if the task queue has an upper limit, should you reject the task,
wait, discard existing tasks, or steal the consumer thread, and have
it execute as it was a worker thread etc. Also the JMS message would
be acknowledged and thus consumed from the JMS broker point of view.
But they are stored in memory in the threads DSL task queue. So in
case of a crash, or a shutdown, you would have to wait for all those
tasks to complete. All together you have a more complicated situation.
You can of course customize the threads DSL to have a lower task
queue. In fact you would be able to have no task queue at all (size of
0) but then you have the overhead of handing over the message from the
JMS consumer to a thread from the threads DSL etc.
Regardless what the current logic in the JmsConsumer is fully
synchronous so even if you use threads, the JmsConsumer would block.
So we would have to improve the source code to support this.
And in the future, eg possible with Camel 3.0+ we will support
asynchronous transactions as well, then this becomes even more
interesting.
> Christian
>
>
> Am 09.08.2011 14:25, schrieb Claus Ibsen:
>>
>> Hi
>>
>> See ticket
>> https://issues.apache.org/jira/browse/CAMEL-3632
>>
>> Take a look at the following routes
>>
>> from("activemq:queue:inbox?synchronous=true")
>> .to("jetty:http://0.0.0.0:9432/myapp")
>> .to("log:result?groupSize=10", "mock:result");
>>
>> from("jetty:http://0.0.0.0:9432/myapp")
>> .delay(100)
>> .transform(body().prepend("Bye "));
>>
>>
>> Its a fairly simple flow JMS -> JETTY -> MOCK
>>
>> The JMS consumer will by default be in synchronous mode. Ticket
>> CAMEL-3632 is to optimize this by supporting asynchronous routing if
>> the JMS consumer is *not* transactional, and *not* sending back a
>> reply message.
>>
>> So the scenarios is for high performance one way messaging.
>>
>> The 2nd route above simulate a little processing time, which takes 100
>> millis.
>>
>> If we run a test where we send 1000 messages to the queue. How long
>> time would it take to process those messages synchronously? Of course
>> roughly 1000 * 0.1 sec = 100 sec. On my laptop it takes:
>> Took: 1 minute (105393 millis)
>>
>> Now imagine we set synchronous=false on the JMS consumer so it runs
>> asynchronously. How fast would it then go? Well lets try
>> Took: 10.742 seconds (10742 millis)
>>
>> That is a lot fast, well of couse its about 10x faster, since the JMS
>> consumer is not blocked waiting for the JETTY reply.
>> However the messages is now not processed in sequence, that is the cost.
>>
>> To note in both scenarios this was done with 1 JMS consumer thread. I
>> did not enable concurrentConsumers. So with the async routing engine
>> we could scale and utilize the CPU resources better.
>>
>> This seems like a great addition. And if we add this, we should add
>> some documentation with the pros/cons for using this.
>>
>> My question is what would a sensitive default setting be? Should we
>> let the JMS consumer be kept as synchronous by default, to keep it
>> fully backwards compatible. Then end users must manually set
>> synchronous=false on the JMS endpoint to enable that.
>>
>> Any thoughts?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: [DISCUSS] - JmsConsumer support for asynchronous routing engine - Higher scalability
Posted by Zbarcea Hadrian <hz...@gmail.com>.
Agree. I think we went overboard in supporting the same scenarios in multiple ways. What is convenient syntactic sugar for some is a source of confusion for many.
My $0.02,
Hadrian
On Aug 9, 2011, at 12:20 PM, Christian Schneider wrote:
> Hi Claus,
>
> without changing anything in camel wouldn´t a simple change in the route also work?
>
> from("activemq:queue:inbox?synchronous=true")
> .threads()
> .to("jetty:http://0.0.0.0:9432/myapp")
> .to("log:result?groupSize=10", "mock:result");
>
> So I ask myself if we really need async support in the jms consumer if we can support the scenario using a well known dsl element.
>
> Christian
>
>
> Am 09.08.2011 14:25, schrieb Claus Ibsen:
>> Hi
>>
>> See ticket
>> https://issues.apache.org/jira/browse/CAMEL-3632
>>
>> Take a look at the following routes
>>
>> from("activemq:queue:inbox?synchronous=true")
>> .to("jetty:http://0.0.0.0:9432/myapp")
>> .to("log:result?groupSize=10", "mock:result");
>>
>> from("jetty:http://0.0.0.0:9432/myapp")
>> .delay(100)
>> .transform(body().prepend("Bye "));
>>
>>
>> Its a fairly simple flow JMS -> JETTY -> MOCK
>>
>> The JMS consumer will by default be in synchronous mode. Ticket
>> CAMEL-3632 is to optimize this by supporting asynchronous routing if
>> the JMS consumer is *not* transactional, and *not* sending back a
>> reply message.
>>
>> So the scenarios is for high performance one way messaging.
>>
>> The 2nd route above simulate a little processing time, which takes 100 millis.
>>
>> If we run a test where we send 1000 messages to the queue. How long
>> time would it take to process those messages synchronously? Of course
>> roughly 1000 * 0.1 sec = 100 sec. On my laptop it takes:
>> Took: 1 minute (105393 millis)
>>
>> Now imagine we set synchronous=false on the JMS consumer so it runs
>> asynchronously. How fast would it then go? Well lets try
>> Took: 10.742 seconds (10742 millis)
>>
>> That is a lot fast, well of couse its about 10x faster, since the JMS
>> consumer is not blocked waiting for the JETTY reply.
>> However the messages is now not processed in sequence, that is the cost.
>>
>> To note in both scenarios this was done with 1 JMS consumer thread. I
>> did not enable concurrentConsumers. So with the async routing engine
>> we could scale and utilize the CPU resources better.
>>
>> This seems like a great addition. And if we add this, we should add
>> some documentation with the pros/cons for using this.
>>
>> My question is what would a sensitive default setting be? Should we
>> let the JMS consumer be kept as synchronous by default, to keep it
>> fully backwards compatible. Then end users must manually set
>> synchronous=false on the JMS endpoint to enable that.
>>
>> Any thoughts?
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> --
> Christian Schneider
> http://www.liquid-reality.de
>
> Open Source Architect
> http://www.talend.com
>
Re: [DISCUSS] - JmsConsumer support for asynchronous routing engine
- Higher scalability
Posted by Christian Schneider <ch...@die-schneider.net>.
Hi Claus,
without changing anything in camel wouldn´t a simple change in the route
also work?
from("activemq:queue:inbox?synchronous=true")
.threads()
.to("jetty:http://0.0.0.0:9432/myapp")
.to("log:result?groupSize=10", "mock:result");
So I ask myself if we really need async support in the jms consumer if
we can support the scenario using a well known dsl element.
Christian
Am 09.08.2011 14:25, schrieb Claus Ibsen:
> Hi
>
> See ticket
> https://issues.apache.org/jira/browse/CAMEL-3632
>
> Take a look at the following routes
>
> from("activemq:queue:inbox?synchronous=true")
> .to("jetty:http://0.0.0.0:9432/myapp")
> .to("log:result?groupSize=10", "mock:result");
>
> from("jetty:http://0.0.0.0:9432/myapp")
> .delay(100)
> .transform(body().prepend("Bye "));
>
>
> Its a fairly simple flow JMS -> JETTY -> MOCK
>
> The JMS consumer will by default be in synchronous mode. Ticket
> CAMEL-3632 is to optimize this by supporting asynchronous routing if
> the JMS consumer is *not* transactional, and *not* sending back a
> reply message.
>
> So the scenarios is for high performance one way messaging.
>
> The 2nd route above simulate a little processing time, which takes 100 millis.
>
> If we run a test where we send 1000 messages to the queue. How long
> time would it take to process those messages synchronously? Of course
> roughly 1000 * 0.1 sec = 100 sec. On my laptop it takes:
> Took: 1 minute (105393 millis)
>
> Now imagine we set synchronous=false on the JMS consumer so it runs
> asynchronously. How fast would it then go? Well lets try
> Took: 10.742 seconds (10742 millis)
>
> That is a lot fast, well of couse its about 10x faster, since the JMS
> consumer is not blocked waiting for the JETTY reply.
> However the messages is now not processed in sequence, that is the cost.
>
> To note in both scenarios this was done with 1 JMS consumer thread. I
> did not enable concurrentConsumers. So with the async routing engine
> we could scale and utilize the CPU resources better.
>
> This seems like a great addition. And if we add this, we should add
> some documentation with the pros/cons for using this.
>
> My question is what would a sensitive default setting be? Should we
> let the JMS consumer be kept as synchronous by default, to keep it
> fully backwards compatible. Then end users must manually set
> synchronous=false on the JMS endpoint to enable that.
>
> Any thoughts?
>
>
>
>
>
>
>
>
>
--
Christian Schneider
http://www.liquid-reality.de
Open Source Architect
http://www.talend.com