You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by bwtaylor <br...@yahoo.com> on 2009/06/25 10:08:48 UTC
Faster!!
I've been doing the simplest possible performance/benchmark test on Camel. I
took a stock activemq 5.2 with camel 1.5 and the example.A to example.B
route, bumped up the queue memory to 50m and ran
ant producer -Dmax=10000 -Dverbose=false -Dsubject=example.A
from the example directory. It took about 38 seconds for the camel route to
finish handing them to example.B.
Next I updated to camel 2.0-M2, I ran the same A to B route in about 38
seconds again.
2.0-M2, with no persistence store finished in 28 seconds.
2.0-M2, with no persistence store and a route of activemq:example.A to
browse:example.B took 24 seconds.
A control load with no persistence took 6 seconds: ant producer -Dmax=10000
-Dverbose=false -Dsubject=example.C
Assuming the browse endpoint write is negligible, this leaves 24-6=18
seconds for activemq to hand messages to camel and camel to ask it's next
endpoint to take them. Presumably the activemq dequeue is no worse than a
write, so camel is introducing 12 seconds to move 10k messages.
Unfortunately, for one of operations I'm working on right now, a complex
event processing problem where I'm using esper, Camel seems to be an
insurmountable speed bottleneck for me. Esper is rocking fast and in my
testing I was pumping 20M pojos in and getting 4M out in 25 seconds. It
looks impossible to get more than about 800 messages per second through
Camel, regardless of what you connect to. This is really annoying, because
camel makes writing esper glue code extremely nice!
Based on the numbers above, camel actually seems comparable to adding
activemq persistence. That doesn't seem reasonable: Camel's speed for simple
from-to routes seems to be the wrong order of magnitude. When I've profiled
in the past, I've seen TypeConverter stuff showing up as the bottleneck. I
don't know if that's still the case or not, but it's a good example where
speed should be able to be substituted for cleverness. I'd love to see Camel
add a set of benchmarks, and publish the numbers every release. On the
prominent benchmarks, I'd love to see every release of Camel having a ticket
or two addressing the top bottlenecks. Surely I'm not the only user out
there interested in seeing a Camel get Faster.
--
View this message in context: http://www.nabble.com/Faster%21%21-tp24198880p24198880.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Faster!!
Posted by Hadrian Zbarcea <hz...@gmail.com>.
Bryan,
Many thanks for sharing your experience. We did not get a lot of
feedback/comments in the recent months about the performance
characteristics of Camel 2.0. We are committed to listening *very*
carefully to what the community has to say and I encourage you to
share as much as you can from your test results and any suggestions
you might have.
We are in the closedown phase of 2.0, we want to make 2.0 much better
and your voice is important. We are quite flexible and we'll squeeze
2.1 improvements in 2.0 whenever possible. The goal is to not delay
2.0 unnecessarily. And obviously we appreciate all the help we can get.
Thanks again,
Hadrian
On Jun 25, 2009, at 2:14 PM, bwtaylor wrote:
>
> Thanks for the thoughtful answer! CAMEL-1078 looks like a good
> start, and I
> see you pulled it in to 2.0.0 (much appreciated!!), so it seems
> progress is
> coming. I also note that CAMEL-963 depends on 1078 and asks for JMS
> to JMS
> handoffs to be comparable to native JMS. That's slated for 2.1. This
> makes
> me happy! More like this!
>
> As far as design tradeoffs, I completely understand. I'd recommend the
> approach that "most" things be simple, and the few hard things
> should be
> possible. So for example, if I know what types my messages should
> be, let me
> tell camel not to use its cleverness and to instead go fast. Your
> suggestion
> on turning error handling and JMX off is another good one. When I
> get a
> chance I play with these and post the results.
>
> I do think that the ability to perform at speeds appropriate to the
> things
> being integrated is an important aspect of "making integration
> easier". Use
> cases like mine where I need 5k messages/second are still somewhat
> rare, but
> they'll be less rare over time, especially when you look at high
> throughput
> components like esper and mina.
>
> I'll also see if I can hook up a profiler and see if I can provide
> some
> vision on what the bottlenecks I see are. If 1078 lands in a
> snapshot I can
> try it out to and report back.
> --
> View this message in context: http://www.nabble.com/Faster%21%21-tp24198880p24208422.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Faster!!
Posted by Claus Ibsen <cl...@gmail.com>.
This discussion about IN, OUT, FAULT is moved to the topic we current
have on the dev forum.
This topic should be used for performance talk.
On Thu, Jul 9, 2009 at 7:44 PM, Claus Ibsen<cl...@gmail.com> wrote:
> On Thu, Jul 9, 2009 at 4:50 PM, Hadrian Zbarcea<hz...@gmail.com> wrote:
>> I hardly see a reason for the IN message to be mutable. Shouldn't it be
>> read-only to start with?
>> In my mind the IN is what you've been told, it's history, cannot be changed.
>> If you want to echo something slightly different, you can always produce
>> the OUT you want (modified copy of the IN).
>
> That doesnt really work with the message exchange patterns. If you
> send in an InOnly and do some processing steps on the exchange, you
> can still mutate the IN message.
>
> If it was a InOut then you could advocate that IN is the original IN
> and cant mutate. And that you should alter the OUT. But then you
> cannot reuse the same processors as from InOnly.
>
>
> I cannot a this time of the hours see a good and concise model for
> this. The current behavior is the best we got at this time.
>
>
>
>
>>
>> +1 on both counts. Properties are used in quite a few places now though,
>> mostly for components that use bindings, like Jms, File, Mail, etc. However
>> I don't think copying properties is very expensive, plus we could optimize
>> that a bit in 2.x.
>>
>> Hadrian
>>
>> On Jul 9, 2009, at 6:35 AM, James Strachan wrote:
>>
>>> 2009/7/9 Claus Ibsen <cl...@gmail.com>:
>>>>
>>>> Hi
>>>>
>>>> See this wiki page for ideas on performance optimizations
>>>> http://camel.apache.org/camel-2x-speed-optimizations.html
>>>
>>> Copying Messages seems to be the big cost (particularly all those headers
>>> etc).
>>>
>>> I wonder if we should use in the Pipeline a ReadOnlyMessageFacade
>>> which prevents mutation of the underlying message, so the same Message
>>> can be reused (no need to make defensive copies first) and can be
>>> passed into next steps in the chain if an OUT message is not created?
>>>
>>> I'm also wondering how often we really use Exchange properties; seems
>>> we mostly use message headers; in which case creating new Exchange
>>> instances should be quite cheap - as we can just pass the IN by value
>>> each time?
>>>
>>> --
>>> James
>>> -------
>>> http://macstrac.blogspot.com/
>>>
>>> Open Source Integration
>>> http://fusesource.com/
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Faster!!
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Jul 9, 2009 at 4:50 PM, Hadrian Zbarcea<hz...@gmail.com> wrote:
> I hardly see a reason for the IN message to be mutable. Shouldn't it be
> read-only to start with?
> In my mind the IN is what you've been told, it's history, cannot be changed.
> If you want to echo something slightly different, you can always produce
> the OUT you want (modified copy of the IN).
That doesnt really work with the message exchange patterns. If you
send in an InOnly and do some processing steps on the exchange, you
can still mutate the IN message.
If it was a InOut then you could advocate that IN is the original IN
and cant mutate. And that you should alter the OUT. But then you
cannot reuse the same processors as from InOnly.
I cannot a this time of the hours see a good and concise model for
this. The current behavior is the best we got at this time.
>
> +1 on both counts. Properties are used in quite a few places now though,
> mostly for components that use bindings, like Jms, File, Mail, etc. However
> I don't think copying properties is very expensive, plus we could optimize
> that a bit in 2.x.
>
> Hadrian
>
> On Jul 9, 2009, at 6:35 AM, James Strachan wrote:
>
>> 2009/7/9 Claus Ibsen <cl...@gmail.com>:
>>>
>>> Hi
>>>
>>> See this wiki page for ideas on performance optimizations
>>> http://camel.apache.org/camel-2x-speed-optimizations.html
>>
>> Copying Messages seems to be the big cost (particularly all those headers
>> etc).
>>
>> I wonder if we should use in the Pipeline a ReadOnlyMessageFacade
>> which prevents mutation of the underlying message, so the same Message
>> can be reused (no need to make defensive copies first) and can be
>> passed into next steps in the chain if an OUT message is not created?
>>
>> I'm also wondering how often we really use Exchange properties; seems
>> we mostly use message headers; in which case creating new Exchange
>> instances should be quite cheap - as we can just pass the IN by value
>> each time?
>>
>> --
>> James
>> -------
>> http://macstrac.blogspot.com/
>>
>> Open Source Integration
>> http://fusesource.com/
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Faster!!
Posted by Hadrian Zbarcea <hz...@gmail.com>.
I hardly see a reason for the IN message to be mutable. Shouldn't it
be read-only to start with?
In my mind the IN is what you've been told, it's history, cannot be
changed. If you want to echo something slightly different, you can
always produce the OUT you want (modified copy of the IN).
+1 on both counts. Properties are used in quite a few places now
though, mostly for components that use bindings, like Jms, File, Mail,
etc. However I don't think copying properties is very expensive, plus
we could optimize that a bit in 2.x.
Hadrian
On Jul 9, 2009, at 6:35 AM, James Strachan wrote:
> 2009/7/9 Claus Ibsen <cl...@gmail.com>:
>> Hi
>>
>> See this wiki page for ideas on performance optimizations
>> http://camel.apache.org/camel-2x-speed-optimizations.html
>
> Copying Messages seems to be the big cost (particularly all those
> headers etc).
>
> I wonder if we should use in the Pipeline a ReadOnlyMessageFacade
> which prevents mutation of the underlying message, so the same Message
> can be reused (no need to make defensive copies first) and can be
> passed into next steps in the chain if an OUT message is not created?
>
> I'm also wondering how often we really use Exchange properties; seems
> we mostly use message headers; in which case creating new Exchange
> instances should be quite cheap - as we can just pass the IN by value
> each time?
>
> --
> James
> -------
> http://macstrac.blogspot.com/
>
> Open Source Integration
> http://fusesource.com/
Re: Faster!!
Posted by James Strachan <ja...@gmail.com>.
2009/7/9 Claus Ibsen <cl...@gmail.com>:
> Hi
>
> See this wiki page for ideas on performance optimizations
> http://camel.apache.org/camel-2x-speed-optimizations.html
Copying Messages seems to be the big cost (particularly all those headers etc).
I wonder if we should use in the Pipeline a ReadOnlyMessageFacade
which prevents mutation of the underlying message, so the same Message
can be reused (no need to make defensive copies first) and can be
passed into next steps in the chain if an OUT message is not created?
I'm also wondering how often we really use Exchange properties; seems
we mostly use message headers; in which case creating new Exchange
instances should be quite cheap - as we can just pass the IN by value
each time?
--
James
-------
http://macstrac.blogspot.com/
Open Source Integration
http://fusesource.com/
Re: Faster!!
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
See this wiki page for ideas on performance optimizations
http://camel.apache.org/camel-2x-speed-optimizations.html
On Mon, Jun 29, 2009 at 4:23 AM, bwtaylor<br...@yahoo.com> wrote:
>
> I like the idea of mode=fast vs mode=smart and of this mode being easily
> tweaked in terms of what features are in it. And this seems like a good page
> for a wiki. The one thing I'd recommend is having some basic benchmarks that
> you do with every release, perhaps in different configurations, and that
> users could run as a system validation.
>
> As far as payloads, for my application they will take several forms, but all
> are small. I parse a single line of text (eg from a log file or a text
> formatted socket), turn it into a small POJO, let esper work some magic on
> it, and I spit out correlation events probably in XML.
> --
> View this message in context: http://www.nabble.com/Faster%21%21-tp24198880p24247565.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Faster!!
Posted by bwtaylor <br...@yahoo.com>.
I like the idea of mode=fast vs mode=smart and of this mode being easily
tweaked in terms of what features are in it. And this seems like a good page
for a wiki. The one thing I'd recommend is having some basic benchmarks that
you do with every release, perhaps in different configurations, and that
users could run as a system validation.
As far as payloads, for my application they will take several forms, but all
are small. I parse a single line of text (eg from a log file or a text
formatted socket), turn it into a small POJO, let esper work some magic on
it, and I spit out correlation events probably in XML.
--
View this message in context: http://www.nabble.com/Faster%21%21-tp24198880p24247565.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Faster!!
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
On Thu, Jun 25, 2009 at 8:14 PM, bwtaylor<br...@yahoo.com> wrote:
>
> Thanks for the thoughtful answer! CAMEL-1078 looks like a good start, and I
> see you pulled it in to 2.0.0 (much appreciated!!), so it seems progress is
> coming. I also note that CAMEL-963 depends on 1078 and asks for JMS to JMS
> handoffs to be comparable to native JMS. That's slated for 2.1. This makes
> me happy! More like this!
>
> As far as design tradeoffs, I completely understand. I'd recommend the
> approach that "most" things be simple, and the few hard things should be
> possible. So for example, if I know what types my messages should be, let me
> tell camel not to use its cleverness and to instead go fast. Your suggestion
> on turning error handling and JMX off is another good one. When I get a
> chance I play with these and post the results.
Another one, in trunk the stream cache is also disabled by default, so
that gives another "little boost".
>
> I do think that the ability to perform at speeds appropriate to the things
> being integrated is an important aspect of "making integration easier". Use
> cases like mine where I need 5k messages/second are still somewhat rare, but
> they'll be less rare over time, especially when you look at high throughput
> components like esper and mina.
Yeah could be nice with some "mode=speed" etc. when you want to go as
fast as possible.
Maybe we should have a wiki page for tips to improve performance or
how to configure Camel for this use case.
And well maybe in the future we could have some easy configuration for
this with a "profile/personality/mode" or whatever it could be named
to set up a preference for this particular route.
Anyway JMS and MINA routing with high speed would be on the radar as
high priority components to be as fast as possible.
BTW: The payloads you need yo send in real life with kind are they? Eg
small message, medium sized message, or big messages?
Are they byte[], stream based, text based or Objects?
And if you need a little content based routing, it could be faster to
have data as headers and use header as predicates for routing instead
of having
to look inside the body payload that is bigger.
>
> I'll also see if I can hook up a profiler and see if I can provide some
> vision on what the bottlenecks I see are. If 1078 lands in a snapshot I can
> try it out to and report back.
Cool that would be awesome.
And I guess when MINA 2.0 is out we could consider upgrading to it, if
its better and faster than 1.1.x
> --
> View this message in context: http://www.nabble.com/Faster%21%21-tp24198880p24208422.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: Faster!!
Posted by bwtaylor <br...@yahoo.com>.
Thanks for the thoughtful answer! CAMEL-1078 looks like a good start, and I
see you pulled it in to 2.0.0 (much appreciated!!), so it seems progress is
coming. I also note that CAMEL-963 depends on 1078 and asks for JMS to JMS
handoffs to be comparable to native JMS. That's slated for 2.1. This makes
me happy! More like this!
As far as design tradeoffs, I completely understand. I'd recommend the
approach that "most" things be simple, and the few hard things should be
possible. So for example, if I know what types my messages should be, let me
tell camel not to use its cleverness and to instead go fast. Your suggestion
on turning error handling and JMX off is another good one. When I get a
chance I play with these and post the results.
I do think that the ability to perform at speeds appropriate to the things
being integrated is an important aspect of "making integration easier". Use
cases like mine where I need 5k messages/second are still somewhat rare, but
they'll be less rare over time, especially when you look at high throughput
components like esper and mina.
I'll also see if I can hook up a profiler and see if I can provide some
vision on what the bottlenecks I see are. If 1078 lands in a snapshot I can
try it out to and report back.
--
View this message in context: http://www.nabble.com/Faster%21%21-tp24198880p24208422.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Faster!!
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Thanks a lot for sharing this great benchmark with us.
Camel was not designed with ultra fast performance in mind. It was
after all making integration easier.
So there is of course a trade off when Camel does the routing for you.
However we are paving the road in post Camel 2.0 timeframe to have the
underlying architecture refactored allowing us to improve the
performance.
Hadrian is currently working on this. See ticket:
https://issues.apache.org/activemq/browse/CAMEL-1078
Yes the type converter had an issue in Camel 1.5.0 that impacted performance.
We have address this since.
There could be a few minor performance gains having some eager checks
before resorting down the type converter chain.
We have also thought of allowing Camel to bootstrap all its
components/converters etc. on startup so it can do pre check and have
it loaded.
Loading all the type converters do take some time.
A few minor tips that could help performance.
1)
Disable error handling. Configure Camel to use the NoErrorHandler.
We should probably have a better switch to disable it instead of
having a noop error handler woven in the route.
That will be optimized when we get the Channel logic being more
intelligent and dynamic.
2)
Disable JMX as it instruments and measure each and every message being routed
3)
Does Camel log alot while you test? I assume INFO level logging does
not produce noise in the log?
4)
And then the Spring JMS listener itself can be optimized.
http://static.springframework.org/spring/docs/2.5.x/api/org/springframework/jms/listener/DefaultMessageListenerContainer.html#setCacheLevel(int)
We love contributions and help. If anyone having time to sit down with
a profiler and check bottlenecks and contention points with Camel that
would be awesome.
In camel-core there is a very simply unit test that outputs a simple
performance on a route: RoutePerformanceTest
Can be used as a start to profile and check spots.
On Thu, Jun 25, 2009 at 10:08 AM, bwtaylor <br...@yahoo.com> wrote:
>
> I've been doing the simplest possible performance/benchmark test on Camel. I
> took a stock activemq 5.2 with camel 1.5 and the example.A to example.B
> route, bumped up the queue memory to 50m and ran
> ant producer -Dmax=10000 -Dverbose=false -Dsubject=example.A
> from the example directory. It took about 38 seconds for the camel route to
> finish handing them to example.B.
>
> Next I updated to camel 2.0-M2, I ran the same A to B route in about 38
> seconds again.
> 2.0-M2, with no persistence store finished in 28 seconds.
> 2.0-M2, with no persistence store and a route of activemq:example.A to
> browse:example.B took 24 seconds.
> A control load with no persistence took 6 seconds: ant producer -Dmax=10000
> -Dverbose=false -Dsubject=example.C
>
> Assuming the browse endpoint write is negligible, this leaves 24-6=18
> seconds for activemq to hand messages to camel and camel to ask it's next
> endpoint to take them. Presumably the activemq dequeue is no worse than a
> write, so camel is introducing 12 seconds to move 10k messages.
> Unfortunately, for one of operations I'm working on right now, a complex
> event processing problem where I'm using esper, Camel seems to be an
> insurmountable speed bottleneck for me. Esper is rocking fast and in my
> testing I was pumping 20M pojos in and getting 4M out in 25 seconds. It
> looks impossible to get more than about 800 messages per second through
> Camel, regardless of what you connect to. This is really annoying, because
> camel makes writing esper glue code extremely nice!
>
> Based on the numbers above, camel actually seems comparable to adding
> activemq persistence. That doesn't seem reasonable: Camel's speed for simple
> from-to routes seems to be the wrong order of magnitude. When I've profiled
> in the past, I've seen TypeConverter stuff showing up as the bottleneck. I
> don't know if that's still the case or not, but it's a good example where
> speed should be able to be substituted for cleverness. I'd love to see Camel
> add a set of benchmarks, and publish the numbers every release. On the
> prominent benchmarks, I'd love to see every release of Camel having a ticket
> or two addressing the top bottlenecks. Surely I'm not the only user out
> there interested in seeing a Camel get Faster.
> --
> View this message in context: http://www.nabble.com/Faster%21%21-tp24198880p24198880.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
Apache Camel Committer
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus