You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@camel.apache.org by "Arkadi Shishlov (JIRA)" <ji...@apache.org> on 2011/05/24 16:25:47 UTC
[jira] [Created] (CAMEL-4009) Fast @Converter loading in uber-jar
Fast @Converter loading in uber-jar
-----------------------------------
Key: CAMEL-4009
URL: https://issues.apache.org/jira/browse/CAMEL-4009
Project: Camel
Issue Type: Improvement
Components: camel-core
Reporter: Arkadi Shishlov
Priority: Minor
Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff
Currently, the @Converter converters are loaded as follows
{noformat}
for package in TypeConverter meta-inf resource
get package url from classloader
if url is jar
for entries in jar
if entry match package
check annotation
{noformat}
This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
The attached patch changes loop into:
{noformat}
for package in TypeConverter meta-inf resource
get package url from classloader
create map url -> package names
for url in distinct url
if jar
for entries in jar
for package in packages
if entry match package
check annotation
{noformat}
With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Arkadi Shishlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arkadi Shishlov updated CAMEL-4009:
-----------------------------------
Attachment: apache-camel-r1128875-fast-TypeConverter-loading-in-uber-jar-try2.diff
After thinking a little bit I implemented in-memory cache of jar entries names via SoftRefence in loadImplementationsInJar(). Please see attached path \-try2.
The memory will be release by GC eventually. I'm not sure the PackageScanClassResolver meant to be thread-safe or not - current usage is single threaded, but anyway, memory ordering is protected by volatile and hash is synchronized to be on the safe side.
The peformance is good - same 2 seconds as with previous patch vs 15 sec of original approach.
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.7.0
> Reporter: Arkadi Shishlov
> Priority: Minor
> Labels: uber-jar
> Fix For: Future
>
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff, apache-camel-r1128875-fast-TypeConverter-loading-in-uber-jar-try2.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen updated CAMEL-4009:
-------------------------------
Affects Version/s: 2.7.0
Fix Version/s: Future
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.7.0
> Reporter: Arkadi Shishlov
> Priority: Minor
> Labels: uber-jar
> Fix For: Future
>
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040762#comment-13040762 ]
Claus Ibsen commented on CAMEL-4009:
------------------------------------
Its the pluggable class resolvers
http://camel.apache.org/pluggable-class-resolvers.html
There is a jboss component for jboss5 and jboss6 at
http://code.google.com/a/apache-extras.org/p/camel-extra/?redir=1
And there is a camel-eclipse in the distro, but we can adjust it if really needed
http://camel.apache.org/camel-eclipse.html
And people could potentially have extended the DefaultPackageScanClassResolver and override the protected methods. So we just have to be a little careful.
So maybe take a 2nd look and try to be as compliant as possible. And polish the code, so its not ugly. Then we can take a 2nd look.
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.7.0
> Reporter: Arkadi Shishlov
> Priority: Minor
> Labels: uber-jar
> Fix For: Future
>
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-4009.
--------------------------------
Resolution: Fixed
Fix Version/s: (was: Future)
2.8.0
Thanks for the patch. I polished it a bit.
Can you test the code on trunk to see if that speedup your uber JAR?
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.7.0
> Reporter: Arkadi Shishlov
> Assignee: Claus Ibsen
> Priority: Minor
> Labels: uber-jar
> Fix For: 2.8.0
>
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff, apache-camel-r1128875-fast-TypeConverter-loading-in-uber-jar-try2.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen reassigned CAMEL-4009:
----------------------------------
Assignee: Claus Ibsen
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.7.0
> Reporter: Arkadi Shishlov
> Assignee: Claus Ibsen
> Priority: Minor
> Labels: uber-jar
> Fix For: Future
>
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff, apache-camel-r1128875-fast-TypeConverter-loading-in-uber-jar-try2.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040621#comment-13040621 ]
Claus Ibsen commented on CAMEL-4009:
------------------------------------
We cannot break the API as there are 3rd party type converter loaders that depend on the current API.
So if you can find a way without breaking the API we may consider the patch. And yes we prefer patches that have been polished and documented.
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.7.0
> Reporter: Arkadi Shishlov
> Priority: Minor
> Labels: uber-jar
> Fix For: Future
>
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Claus Ibsen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Claus Ibsen resolved CAMEL-4009.
--------------------------------
Resolution: Fixed
Okay a better impl is in trunk now
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.7.0
> Reporter: Arkadi Shishlov
> Assignee: Claus Ibsen
> Priority: Minor
> Labels: uber-jar
> Fix For: 2.8.0
>
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff, apache-camel-r1128875-fast-TypeConverter-loading-in-uber-jar-try2.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Arkadi Shishlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040667#comment-13040667 ]
Arkadi Shishlov commented on CAMEL-4009:
----------------------------------------
bq. We cannot break the API as there are 3rd party type converter loaders that depend on the current API.
Could you please point me to them so I could evaluate the impact?
bq. So if you can find a way without breaking the API we may consider the patch. And yes we prefer patches that have been polished and documented.
I believe the public API is not broken. I only changed _protected find()_ signatures and semantic of _customResourceLocator()_ is left intact. I can add original _find(String package)_ if you wish.
Otherwise, is it ok?
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.7.0
> Reporter: Arkadi Shishlov
> Priority: Minor
> Labels: uber-jar
> Fix For: Future
>
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Arkadi Shishlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arkadi Shishlov reopened CAMEL-4009:
------------------------------------
The TRUNK version won't speed up.
You're clearing the cache but is supposed to be kept across different calls to find(). Thats why the jarCache was implemented with class member and SoftRef to release memory after resolver is not used anymore.
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Affects Versions: 2.7.0
> Reporter: Arkadi Shishlov
> Assignee: Claus Ibsen
> Priority: Minor
> Labels: uber-jar
> Fix For: 2.8.0
>
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff, apache-camel-r1128875-fast-TypeConverter-loading-in-uber-jar-try2.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CAMEL-4009) Fast @Converter loading in uber-jar
Posted by "Arkadi Shishlov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CAMEL-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arkadi Shishlov updated CAMEL-4009:
-----------------------------------
Attachment: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff
> Fast @Converter loading in uber-jar
> -----------------------------------
>
> Key: CAMEL-4009
> URL: https://issues.apache.org/jira/browse/CAMEL-4009
> Project: Camel
> Issue Type: Improvement
> Components: camel-core
> Reporter: Arkadi Shishlov
> Priority: Minor
> Labels: uber-jar
> Attachments: apache-camel-r1127037-fast-TypeConverter-loading-in-uber-jar.diff
>
>
> Currently, the @Converter converters are loaded as follows
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> if url is jar
> for entries in jar
> if entry match package
> check annotation
> {noformat}
> This is very inefficient approach when application is packaged into uber-jar as expensive iteration is performed multiple times over same jar.
> The attached patch changes loop into:
> {noformat}
> for package in TypeConverter meta-inf resource
> get package url from classloader
> create map url -> package names
> for url in distinct url
> if jar
> for entries in jar
> for package in packages
> if entry match package
> check annotation
> {noformat}
> With this patch I was able to cut startup time significantly from 15sec spent on @Converter-s load to 2sec on 45MB 29k entries uber-jar.
> I understand that the solution is not pretty but resolves a very practical problem. Let me know if you want me to polish it and update the javadoc.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira