You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by Christian Müller <ch...@gmail.com> on 2012/08/28 23:07:28 UTC
ESB Performance Testing - Round 6
You may know the last ESB performance test results from round 6 [1].
As you can see, Apache ServiceMix 4.3.0 failed to complete this benchmark.
These tests are still based on the deprecated Apache ServiceMix JBI
components.
Because of this, I implemented the required tests for Apache ServiceMix
4.4.2 and Apache Camel (2.8.5) - instead of using JBI. I hope we can pass
the tests this time...
Feel free to provide any feedback. May you find places where we can make
Apache ServiceMix and/or Apache Camel faster.
You can find my Mercurial repo at [2] and my Git repo at [3].
[1]
http://esbperformance.org/display/comparison/ESB+Performance+Testing+-+Round+6
[2] https://bitbucket.org/muellerc/esbperformance
[3] https://github.com/muellerc/esbperformance
Best,
Christian
--
Re: ESB Performance Testing - Round 6
Posted by Daniel Kulp <dk...@apache.org>.
(responding to both Claus and Christian)
On Aug 29, 2012, at 3:11 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Tue, Aug 28, 2012 at 11:07 PM, Christian Müller
> <ch...@gmail.com> wrote:
>> You may know the last ESB performance test results from round 6 [1].
>> As you can see, Apache ServiceMix 4.3.0 failed to complete this benchmark.
>> These tests are still based on the deprecated Apache ServiceMix JBI
>> components.
>>
>
> First of all I think this test is single eye-sided on the area that
> the people behind the tests,
> is the areas where their solution is as its bests = only WS. And its
> not a general purpose tests to show
> all areas of the ESB. eg its only webservice based!
Well, feel free to propose new test cases. I do know that for Round 7, they want to add a bunch of REST based tests, but no ideas on what they may look like yet.
>> Because of this, I implemented the required tests for Apache ServiceMix
>> 4.4.2 and Apache Camel (2.8.5) - instead of using JBI. I hope we can pass
>> the tests this time…
Well, I would have expected SOME credit for this since you obviously started with much of the work I did for Talend ESB……. ;-)
Also have to be careful about sticking the Apache license header on some of those (particularly the ws-security context.xml). I have no idea what the original license is. I never really asked about that. Likely Apache or Apache compatible, but I really have no idea.
> I guess when the time comes it makes sense to use SMX 4.5.0 with Camel
> 2.10.x and CXF 2.6.x to get the latest stuff.
> Especially since both Camel and CXF in these releases have been
> improved in the streaming areas of XML / CXF stuff.
> I am not sure though how much this applies to the tests where you run
> in MESSAGE mode in camel-cxf.
Not much. :-) A lot of those performance enhancements were done before I flipped the Talend tests over to MESSAGE. Once I flipped them to MESSAGE, it pretty much bypassed most of the work I did optimizing the streaming payload. The ws-security tests would benefit a bit, but not much. The streaming payload stuff is great and certainly helps in a bunch of scenarios, but not this one anymore.
>> Feel free to provide any feedback. May you find places where we can make
>
> As the tests uses big/medium payloads 50kb, 100kb etc. and you have
> enable stream caching in the code, then mind the default threshold for
> spooling to disk is at 64kb limit. And when you spool to disk its
> freaking slow compared to be kept in memory.
> And its using the slow java.io.File API and not any NIO memory mapped
> files etc instead.
>
> You can set an option to increase the threshold, to keep the messages in memory
> http://camel.apache.org/stream-caching.html
Actually, I did look at this and for these tests, it doesn't matter. The only time the tests go to 100K is in the direct proxy case. The other 5 tests top out at 10K. The direct proxy doesn't cache the streams so the setting isn't needed for these tests. It is something to keep an eye on if they add larger tests. (example: between round 5 and 6, I had them add larger ws-security tests as ws-security is important for me and I wanted to see how it behaved as the messages got a little larget)
Dan
> Also I wonder if using OSGi blueprint would be better than the old
> aging spring-dm 1.2.
> And I guess the SMX can be trimmed down in size. Maybe with its
> minimal configuration etc.
> eg to drop all the legacy JBI and whatnot it carries by default.
>
>
>> Apache ServiceMix and/or Apache Camel faster.
>> You can find my Mercurial repo at [2] and my Git repo at [3].
>>
>> [1]
>> http://esbperformance.org/display/comparison/ESB+Performance+Testing+-+Round+6
>> [2] https://bitbucket.org/muellerc/esbperformance
>> [3] https://github.com/muellerc/esbperformance
>>
>> Best,
>> Christian
>>
>> --
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
--
Daniel Kulp
dkulp@apache.org - http://dankulp.com/blog
Talend Community Coder - http://coders.talend.com
Re: ESB Performance Testing - Round 6
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Aug 28, 2012 at 11:07 PM, Christian Müller
<ch...@gmail.com> wrote:
> You may know the last ESB performance test results from round 6 [1].
> As you can see, Apache ServiceMix 4.3.0 failed to complete this benchmark.
> These tests are still based on the deprecated Apache ServiceMix JBI
> components.
>
First of all I think this test is single eye-sided on the area that
the people behind the tests,
is the areas where their solution is as its bests = only WS. And its
not a general purpose tests to show
all areas of the ESB. eg its only webservice based!
> Because of this, I implemented the required tests for Apache ServiceMix
> 4.4.2 and Apache Camel (2.8.5) - instead of using JBI. I hope we can pass
> the tests this time...
I guess when the time comes it makes sense to use SMX 4.5.0 with Camel
2.10.x and CXF 2.6.x to get the latest stuff.
Especially since both Camel and CXF in these releases have been
improved in the streaming areas of XML / CXF stuff.
I am not sure though how much this applies to the tests where you run
in MESSAGE mode in camel-cxf.
> Feel free to provide any feedback. May you find places where we can make
As the tests uses big/medium payloads 50kb, 100kb etc. and you have
enable stream caching in the code, then mind the default threshold for
spooling to disk is at 64kb limit. And when you spool to disk its
freaking slow compared to be kept in memory.
And its using the slow java.io.File API and not any NIO memory mapped
files etc instead.
You can set an option to increase the threshold, to keep the messages in memory
http://camel.apache.org/stream-caching.html
Also I wonder if using OSGi blueprint would be better than the old
aging spring-dm 1.2.
And I guess the SMX can be trimmed down in size. Maybe with its
minimal configuration etc.
eg to drop all the legacy JBI and whatnot it carries by default.
> Apache ServiceMix and/or Apache Camel faster.
> You can find my Mercurial repo at [2] and my Git repo at [3].
>
> [1]
> http://esbperformance.org/display/comparison/ESB+Performance+Testing+-+Round+6
> [2] https://bitbucket.org/muellerc/esbperformance
> [3] https://github.com/muellerc/esbperformance
>
> Best,
> Christian
>
> --
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: ESB Performance Testing - Round 6
Posted by Ruwan Linton <ru...@gmail.com>.
Hi all,
Adding to Asankha..
Replying to Claus, on VTD-XML, UltraESB too is using VTD-XML to do xpath
etc in its enhanced version (due to the same licensing reasons mentioned by
Claus) of the perf tests. In the same manner, we could test a vanilla
servicemix version as well as an enhanced servicemix version with VTD-XML
parsing enabled, as we would like to test the downloadable version an any
enhanced versions separately.
On another note, I saw in this discussion Claus mentioning about, "spooling
into the Disk" with some sort of a streaming cache, for that we could
utilize a RAM-DISK which is already setup on the AMI instance (/tmp/ram)
containing the performance setup (UltraESB is utilizing this for its
file-cache).
We really appreciate any feedback on improving this test suite to support
other ESB scenarios, than the once using HTTP transport (which I believe is
being tested well with WS and REST). Also note that the AMI is open and
anybody can run the tests them selves too, if they have an EC2 account.
(IIRC, Apache has some for Apache projects)
Thanks,
Ruwan
On Fri, Aug 31, 2012 at 12:46 PM, asankha <as...@gmail.com> wrote:
> Hi Christian, Dan and Claus
>
> >And the fact that they use SMX 4.3.0 which is an old release
> (4.3.0/03-Mar-2011 13:25)
> >Since SMX 4.4.0, 4.4.1, and 4.4.2 has been released.
>
> First of all I must apologize for this.. But I visited the ServiceMix
> http://servicemix.apache.org/download.html download page just now, and it
> still tells me "The latest release is the ServiceMix 4.3.0 Release. We
> strongly encourage all users to use this Release" - so I'm sure something
> needs to be corrected somewhere..
>
> We would really like to work with you all to define how the test suite
> should be extended - like Dan said, we made some enhancements this round,
> and expect to bring in JSON/REST test cases. Other transports such as
> File/JMS etc would be more difficult to test - especially for performance -
> but if you could propose clever ways of doing it, we'd certainly like to
> include them. The only reason we do not test large messages for all
> scenarios, and large numbers of messages for high concurrency levels etc,
> is
> just to make the total test run within a decent time across all ESBs
>
> As for the updates submitted by Christian, we will run them ASAP - and get
> back to you. This was our original objective as well when we
> http://www.prweb.com/releases/2012/6/prweb9651239.htm announced Round 6
> and
> asked for updates from any vendors or contributors, and a chance to
> optimize
> and fine tune the configurations before the final tests were run
>
> We did work cooperatively with those who replied to us, and wanted to
> ensure
> a fair benchmark that was also transparent - thus we accepted almost all of
> the suggestions and enhancements that came in - and I'm sure participating
> in the round helped those ESBs find out some areas where they could improve
> as well, which is good.
>
> Again, I'd like to commit that we'd like to cooperate with you on this, and
> work together - to make a better, fair and transparent benchmark for ESBs!
>
> regards
> asankha
>
> AdroitLogic Private Ltd
> http://www.adroitlogic.org
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/ESB-Performance-Testing-Round-6-tp5718215p5718459.html
> Sent from the Camel Development mailing list archive at Nabble.com.
>
--
Ruwan Linton
Member, Apache Software Foundation; http://www.apache.org
Director of Engineering; http://adroitlogic.org
phone: +94 11 251 6899
email: ruwan@adroitlogic.com; cell: +94 77 341 3097
blog: http://blog.ruwan.org
linkedin: http://www.linkedin.com/in/ruwanlinton
google: http://www.google.com/profiles/ruwan.linton
tweet: http://twitter.com/ruwanlinton
Re: ESB Performance Testing - Round 6
Posted by asankha <as...@gmail.com>.
Hi Christian, Dan and Claus
>And the fact that they use SMX 4.3.0 which is an old release
(4.3.0/03-Mar-2011 13:25)
>Since SMX 4.4.0, 4.4.1, and 4.4.2 has been released.
First of all I must apologize for this.. But I visited the ServiceMix
http://servicemix.apache.org/download.html download page just now, and it
still tells me "The latest release is the ServiceMix 4.3.0 Release. We
strongly encourage all users to use this Release" - so I'm sure something
needs to be corrected somewhere..
We would really like to work with you all to define how the test suite
should be extended - like Dan said, we made some enhancements this round,
and expect to bring in JSON/REST test cases. Other transports such as
File/JMS etc would be more difficult to test - especially for performance -
but if you could propose clever ways of doing it, we'd certainly like to
include them. The only reason we do not test large messages for all
scenarios, and large numbers of messages for high concurrency levels etc, is
just to make the total test run within a decent time across all ESBs
As for the updates submitted by Christian, we will run them ASAP - and get
back to you. This was our original objective as well when we
http://www.prweb.com/releases/2012/6/prweb9651239.htm announced Round 6 and
asked for updates from any vendors or contributors, and a chance to optimize
and fine tune the configurations before the final tests were run
We did work cooperatively with those who replied to us, and wanted to ensure
a fair benchmark that was also transparent - thus we accepted almost all of
the suggestions and enhancements that came in - and I'm sure participating
in the round helped those ESBs find out some areas where they could improve
as well, which is good.
Again, I'd like to commit that we'd like to cooperate with you on this, and
work together - to make a better, fair and transparent benchmark for ESBs!
regards
asankha
AdroitLogic Private Ltd
http://www.adroitlogic.org
--
View this message in context: http://camel.465427.n5.nabble.com/ESB-Performance-Testing-Round-6-tp5718215p5718459.html
Sent from the Camel Development mailing list archive at Nabble.com.
Re: ESB Performance Testing - Round 6
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Aug 30, 2012 at 12:20 AM, Christian Müller
<ch...@gmail.com> wrote:
> Hello Claus, hello Dan!
>
> First, Kudos for Dan. I was indeed inspired from your performance tests.
> Thanks for your work.
>
> May be this performance test is single-sided from our point of view (I
> share this view.). But to fail with this tests cause some concerns by my
> managers. They didn't know that the integration solution was implemented
> with the deprecated JBI components. They only notice: ServiceMix (and Fuse
> ESB) failed to complete the test. That's all.
> And because Camel is the first choice integration solution for ServiceMix,
> we can "help out" here and correct the bad picture which some people got,
> after reading this test.
>
Ah yeah I didn't know that either that the old JBI stuff was in use.
And the fact that they use SMX 4.3.0 which is an old release (4.3.0/
03-Mar-2011 13:25)
Since SMX 4.4.0, 4.4.1, and 4.4.2 has been released.
I am not neither sure if they have configured the old JBI apps in an
optimal way, for example
https://bitbucket.org/adroitlogic/esbperformance/src/a6284cd21221/servicemix/proxy/proxy-cxf-su/src/main/resources/cxf-config.xml
... shows a thread pool with max 24 threads, where as some of the
other ESB being tested is using much higher thread pools in the
hundreds.
A few tips to optimize your Camel tests would be to
- disable JMX either totally, or set the performance stats off
(http://camel.apache.org/camel-jmx.html)
- turn of breadcrumb (http://camel.apache.org/mdc-logging.html)
- check logging if something is logging to noisy (albeit at INFO level
it should not be noisy)
- and the stream caching threshold I talked about before
And then I guess there is a some CXF / Jetty stuff that can be tweaked a bit.
I am not sure about how their thread pool settings is configured out
of the box etc.
The pure proxy example, you may use pure Jetty instead of CXF, albeit
that CXF in MESSAGE mode probably is super fast anyway.
And CXF uses Jetty under the hood anyway.
http://camel.apache.org/how-to-use-camel-as-a-http-proxy-between-a-client-and-server.html
Also I wonder if we could optimize the stream cache implementation in Camel to
- use file nio for spooling to disk (allow to use nio and thinks like
memory mapped files)
- avoid copy stream, if input source supports re-readable (eg to set a
mark on the stream, and position back to the mark)
> And I also would like to work on a more "real world" ESB or integration
> framework performance test suite (one task in my long TODO list...). This
> test suite would also test other frequently used protocols like file, http,
> jms, rest or plain tcp/ip and also a few more frequently used EIP's like
> splitter, aggregator, content based router (not only the filter EIP) and
> message translator. It's good to know how good/bad we perform compared to
> the other ones (I know, performance is not the single fact which is
> important, but it's one of a few).
>
> The guys from http://esbperformance.org<http://esbperformance.org/display/comparison/ESB+Performance+Testing+-+Round+6>will
> share the results with me/us before they will run a new performance
> test. This allows us to tweak our solution (with your valuable input). And
> after ServiceMix 4.5.0 is out, I will upgrade the tests to use this version
> of ServiceMix. Until then, I will work on the changes you suggested and see
> how it works. Some of the suggestion are already "implemented" (a minimal
> ServiceMix feature boot path). Hopefully I will find time to play with the
> VTD-XML library. I didn't used it until now.
>
> Best,
> Christian
>
> On Wed, Aug 29, 2012 at 6:48 PM, Claus Ibsen <cl...@gmail.com> wrote:
>
>> Hi
>>
>> For what may be worth it, Camel now supports the VTD-XML library which
>> is a very fast and efficient XPath engine.
>> http://camel.apache.org/vtd-xml
>>
>> Though the camel-vtdxml component is hosted at Camel Extra due GPL
>> licensing of the VTD-XML library itself.
>>
>> It works well for processing/splitting big files etc, where the xpath
>> and xquery can't cope so well (xquery using saxon).
>> I am not sure if any of the ESB's use that for xpath/xquery. But with
>> Camel we can now do that as well.
>>
>> Installing it in OSGi I have not done. Hopefully installing the JAR
>> using wrap would be sufficient.
>>
>>
>>
>> On Tue, Aug 28, 2012 at 11:07 PM, Christian Müller
>> <ch...@gmail.com> wrote:
>> > You may know the last ESB performance test results from round 6 [1].
>> > As you can see, Apache ServiceMix 4.3.0 failed to complete this
>> benchmark.
>> > These tests are still based on the deprecated Apache ServiceMix JBI
>> > components.
>> >
>> > Because of this, I implemented the required tests for Apache ServiceMix
>> > 4.4.2 and Apache Camel (2.8.5) - instead of using JBI. I hope we can pass
>> > the tests this time...
>> > Feel free to provide any feedback. May you find places where we can make
>> > Apache ServiceMix and/or Apache Camel faster.
>> > You can find my Mercurial repo at [2] and my Git repo at [3].
>> >
>> > [1]
>> >
>> http://esbperformance.org/display/comparison/ESB+Performance+Testing+-+Round+6
>> > [2] https://bitbucket.org/muellerc/esbperformance
>> > [3] https://github.com/muellerc/esbperformance
>> >
>> > Best,
>> > Christian
>> >
>> > --
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>>
>
>
>
> --
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Re: ESB Performance Testing - Round 6
Posted by Christian Müller <ch...@gmail.com>.
Hello Claus, hello Dan!
First, Kudos for Dan. I was indeed inspired from your performance tests.
Thanks for your work.
May be this performance test is single-sided from our point of view (I
share this view.). But to fail with this tests cause some concerns by my
managers. They didn't know that the integration solution was implemented
with the deprecated JBI components. They only notice: ServiceMix (and Fuse
ESB) failed to complete the test. That's all.
And because Camel is the first choice integration solution for ServiceMix,
we can "help out" here and correct the bad picture which some people got,
after reading this test.
And I also would like to work on a more "real world" ESB or integration
framework performance test suite (one task in my long TODO list...). This
test suite would also test other frequently used protocols like file, http,
jms, rest or plain tcp/ip and also a few more frequently used EIP's like
splitter, aggregator, content based router (not only the filter EIP) and
message translator. It's good to know how good/bad we perform compared to
the other ones (I know, performance is not the single fact which is
important, but it's one of a few).
The guys from http://esbperformance.org<http://esbperformance.org/display/comparison/ESB+Performance+Testing+-+Round+6>will
share the results with me/us before they will run a new performance
test. This allows us to tweak our solution (with your valuable input). And
after ServiceMix 4.5.0 is out, I will upgrade the tests to use this version
of ServiceMix. Until then, I will work on the changes you suggested and see
how it works. Some of the suggestion are already "implemented" (a minimal
ServiceMix feature boot path). Hopefully I will find time to play with the
VTD-XML library. I didn't used it until now.
Best,
Christian
On Wed, Aug 29, 2012 at 6:48 PM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> For what may be worth it, Camel now supports the VTD-XML library which
> is a very fast and efficient XPath engine.
> http://camel.apache.org/vtd-xml
>
> Though the camel-vtdxml component is hosted at Camel Extra due GPL
> licensing of the VTD-XML library itself.
>
> It works well for processing/splitting big files etc, where the xpath
> and xquery can't cope so well (xquery using saxon).
> I am not sure if any of the ESB's use that for xpath/xquery. But with
> Camel we can now do that as well.
>
> Installing it in OSGi I have not done. Hopefully installing the JAR
> using wrap would be sufficient.
>
>
>
> On Tue, Aug 28, 2012 at 11:07 PM, Christian Müller
> <ch...@gmail.com> wrote:
> > You may know the last ESB performance test results from round 6 [1].
> > As you can see, Apache ServiceMix 4.3.0 failed to complete this
> benchmark.
> > These tests are still based on the deprecated Apache ServiceMix JBI
> > components.
> >
> > Because of this, I implemented the required tests for Apache ServiceMix
> > 4.4.2 and Apache Camel (2.8.5) - instead of using JBI. I hope we can pass
> > the tests this time...
> > Feel free to provide any feedback. May you find places where we can make
> > Apache ServiceMix and/or Apache Camel faster.
> > You can find my Mercurial repo at [2] and my Git repo at [3].
> >
> > [1]
> >
> http://esbperformance.org/display/comparison/ESB+Performance+Testing+-+Round+6
> > [2] https://bitbucket.org/muellerc/esbperformance
> > [3] https://github.com/muellerc/esbperformance
> >
> > Best,
> > Christian
> >
> > --
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
>
--
Re: ESB Performance Testing - Round 6
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
For what may be worth it, Camel now supports the VTD-XML library which
is a very fast and efficient XPath engine.
http://camel.apache.org/vtd-xml
Though the camel-vtdxml component is hosted at Camel Extra due GPL
licensing of the VTD-XML library itself.
It works well for processing/splitting big files etc, where the xpath
and xquery can't cope so well (xquery using saxon).
I am not sure if any of the ESB's use that for xpath/xquery. But with
Camel we can now do that as well.
Installing it in OSGi I have not done. Hopefully installing the JAR
using wrap would be sufficient.
On Tue, Aug 28, 2012 at 11:07 PM, Christian Müller
<ch...@gmail.com> wrote:
> You may know the last ESB performance test results from round 6 [1].
> As you can see, Apache ServiceMix 4.3.0 failed to complete this benchmark.
> These tests are still based on the deprecated Apache ServiceMix JBI
> components.
>
> Because of this, I implemented the required tests for Apache ServiceMix
> 4.4.2 and Apache Camel (2.8.5) - instead of using JBI. I hope we can pass
> the tests this time...
> Feel free to provide any feedback. May you find places where we can make
> Apache ServiceMix and/or Apache Camel faster.
> You can find my Mercurial repo at [2] and my Git repo at [3].
>
> [1]
> http://esbperformance.org/display/comparison/ESB+Performance+Testing+-+Round+6
> [2] https://bitbucket.org/muellerc/esbperformance
> [3] https://github.com/muellerc/esbperformance
>
> Best,
> Christian
>
> --
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen