You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@camel.apache.org by "Les Novell (JIRA)" <ji...@apache.org> on 2012/10/02 00:37:07 UTC
[jira] [Created] (CAMEL-5675) Camel Route Startup Performance Slow
Les Novell created CAMEL-5675:
---------------------------------
Summary: Camel Route Startup Performance Slow
Key: CAMEL-5675
URL: https://issues.apache.org/jira/browse/CAMEL-5675
Project: Camel
Issue Type: Improvement
Components: camel-core
Affects Versions: 2.8.6
Environment: JDK 1.6
Reporter: Les Novell
I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-5675) Camel Route Startup Performance Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-5675:
-------------------------------
Attachment: CAMEL-5675.patch
I have attached a patch which has caching on JMX and IntrospectionSupport.
The most noticeable performance improvements is when JMX is enabled, and if you have large routes.
The big route tests is about cut down from 8s to 4s on startup (looking at the startup time Camel reports itself).
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
> Attachments: CAMEL-5675.patch
>
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5675) Camel Route Startup Performance
Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470065#comment-13470065 ]
Claus Ibsen commented on CAMEL-5675:
------------------------------------
I did change the cache to use weak reference based. So if you have already run some tests, then try again with latest code from trunk.
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
> Fix For: 2.11.0
>
> Attachments: after.png, before.png, CAMEL-5675.patch
>
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5675) Camel Route Startup Performance
Slow
Posted by "David J. M. Karlsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468392#comment-13468392 ]
David J. M. Karlsen commented on CAMEL-5675:
--------------------------------------------
You could maybe use commons which has introspection cache already?
http://commons.apache.org/beanutils/xref/org/apache/commons/beanutils/PropertyUtils.html
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5675) Camel Route Startup Performance
Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470038#comment-13470038 ]
Claus Ibsen commented on CAMEL-5675:
------------------------------------
If you run Camel 2.9+ then in the unit tests you can tell Camel that you use advice with, as said in the green box at
http://camel.apache.org/advicewith.html
That should avoid the restart of routes.
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
> Fix For: 2.11.0
>
> Attachments: after.png, before.png, CAMEL-5675.patch
>
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5675) Camel Route Startup Performance
Slow
Posted by "Les Novell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13469733#comment-13469733 ]
Les Novell commented on CAMEL-5675:
-----------------------------------
Wow, thank you Claus. I was off work yesterday & surprised to come back today and see this issue already addressed :) I will definitely give the changes a try & report back the results.
I agree that the overall performance is generally OK. It was taking 1/2 - 1 sec to initialize our routes. We were also applying advice in order to mock out certain routes, which causes the routes to be stopped and restarted. All this caused a unit test to take 1-2 seconds on average. We try and target our unit tests to run in <1/10 second, since our unit test suite is large (1200+ tests).
Your fix to the Introspection classes should help immensely.
Much Thanks,
Les
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
> Fix For: 2.11.0
>
> Attachments: after.png, before.png, CAMEL-5675.patch
>
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-5675) Camel Route Startup Performance Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-5675:
-------------------------------
Attachment: after.png
after screenshot with patch.
Notice the much lower invocation count, and the self time is also much lower.
Apache Camel (CamelContext: camel-1) started in 2.447 seconds
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
> Attachments: after.png, before.png, CAMEL-5675.patch
>
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5675) Camel Route Startup Performance
Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467758#comment-13467758 ]
Claus Ibsen commented on CAMEL-5675:
------------------------------------
I asume you have JMX enabled as well, as it does a fair bit of introspection as well.
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CAMEL-5675) Camel Route Startup Performance Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen reassigned CAMEL-5675:
----------------------------------
Assignee: Claus Ibsen
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-5675) Camel Route Startup Performance Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-5675:
-------------------------------
Attachment: before.png
Screenshot from profiler with full tracing enabled.
Running the big route test (lowered to 500, instead of 1000 nodes)
Apache Camel (CamelContext: camel-1) started in 4.958 seconds
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
> Attachments: before.png, CAMEL-5675.patch
>
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5675) Camel Route Startup Performance
Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467779#comment-13467779 ]
Claus Ibsen commented on CAMEL-5675:
------------------------------------
Yeah I think we could have caching during startup of CamelContext. Having cache at runtime is a bit more problematic, as you would need to be able to limit the cache size. And as well have soft reference to avoid classloader issues etc.
Its easier to cache during startup only (and during shutdown as well) as we can control when caching is enabled/disable.
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CAMEL-5675) Camel Route Startup Performance Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-5675.
--------------------------------
Resolution: Fixed
Fix Version/s: 2.11.0
Should be faster now, and no longer so high invocation counts.
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
> Fix For: 2.11.0
>
> Attachments: after.png, before.png, CAMEL-5675.patch
>
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5675) Camel Route Startup Performance
Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468516#comment-13468516 ]
Claus Ibsen commented on CAMEL-5675:
------------------------------------
David,
Thanks for the link. There is although already a cache in Camel we can leverage, LRUSoftCache.
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
> Attachments: CAMEL-5675.patch
>
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-5675) Camel Route Startup Performance
Slow
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-5675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468416#comment-13468416 ]
Claus Ibsen commented on CAMEL-5675:
------------------------------------
Well even with some caching it only speedup for very big routes.
Les, can you tell us a bit about your setup? How many tests do you run? Do you have many routes, and are they small/big etc?
And do you have JMX enabled during testing?
For the Camel tests of itself there is hardly a speedup (many small route tests). And most tests have disabled JMX.
For a big route with 1000 nodes and JMX enabled, I could cut down startup time with 1.5-2s. But just wonder who would have such big routes?
> Camel Route Startup Performance Slow
> ------------------------------------
>
> Key: CAMEL-5675
> URL: https://issues.apache.org/jira/browse/CAMEL-5675
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.8.6
> Environment: JDK 1.6
> Reporter: Les Novell
> Assignee: Claus Ibsen
> Labels: performance
>
> I am writing unit tests for Camel and found that each unit test was taking up to a second just to create the Camel routes. That's not very long, but we have a large unit test suite that needs to run quickly.
> I did a performance profile and found that most of the time is going to the method org.apache.camel.util.IntrospectionSupport.getProperties(Object, Map, String). That method, then also calls IntrospectionSupport.isSetter(Method), and just running two unit tests I saw isSettter called 2.5 million times!
> It seems to me that a cache per class of the properties would make a huge performance improvement on Camel route building.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira