You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Ittay Dror (JIRA)" <ji...@codehaus.org> on 2008/07/03 09:06:28 UTC
[jira] Created: (MNG-3647) Maven performance needs improvement
Maven performance needs improvement
-----------------------------------
Key: MNG-3647
URL: http://jira.codehaus.org/browse/MNG-3647
Project: Maven 2
Issue Type: Bug
Affects Versions: 2.0.8
Reporter: Ittay Dror
I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
There are other issues which are harder to tackle:
1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Ittay Dror (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=140381#action_140381 ]
Ittay Dror commented on MNG-3647:
---------------------------------
One more aspect is the amount of empty mojos being executed. Take for example the abdera project. It has 28 simple modules. calling 'compile' creates 78 executions of mojos. Since every execution incurs an overhead, even if the mojo doesn't do anything, this is alot. With more complex projects, where plugins are configured in parent poms but actually do anything in child poms, the overhead is larger.
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Ittay Dror (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=141022#action_141022 ]
Ittay Dror commented on MNG-3647:
---------------------------------
Note: I don't want this issue to become a discussion about a suggestion I made in a comment and not the concrete improvements I wrote in the main description.
About the plugins. Think of a scenario where I write a pom.xml for a module where I know there isn't any src/main/resources folder, and hence nothing for the resources plugin to do. What I would want to be able to do is to write in pom.xml something like:
....
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<disabled>true</disabled>
</plugin>
maven will then remove this plugin from the lifecycle, and with it the overhead of initializing all objects and data just so the 'execute' doesn't do anything.
i fail to see how it goes against pluggability or out-of-the-box. if i do nothing, the plugin runs (maybe print a warning that it has nothing to do and suggest it will be disabled?)
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Milos Kleint (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=140382#action_140382 ]
Milos Kleint commented on MNG-3647:
-----------------------------------
IMHO we shall not optimize for badly configured projects (mojos/plugins shall be configured in pluginManagement section in parent and only added to plugins section of projects that actually make use of it.
I would be more than happy if we could just cut the mojo setup overhead reasonably down.
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Ittay Dror (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=140345#action_140345 ]
Ittay Dror commented on MNG-3647:
---------------------------------
Note: i tried profiling with several profilers. The only one that was not confused by all the class loading and dynamic invocations is JProbe.
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Milos Kleint (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=141017#action_141017 ]
Milos Kleint commented on MNG-3647:
-----------------------------------
your suggestions goes against the pluggability and out-of-the box principles that maven is built on. You can't know if resources-plugin will do anything before you run it. Your suggestion about configuring each plugin might work for you, but it's not a good general option. You can do that even now. Define a custom packaging ("void" for example) that has no mojos associated with lifecycle phases and the configuration on your own for each plugin. However I'm not convinced it's worth the effort. I believe it's the maven core that needs to be optimized first.
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Closed: (MNG-3647) Maven performance needs improvement
Posted by "Jason van Zyl (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason van Zyl closed MNG-3647.
------------------------------
Resolution: Fixed
Fix Version/s: (was: 2.2.x)
3.x
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2 & 3
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
> Fix For: 3.x
>
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Jason van Zyl (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=204608#action_204608 ]
Jason van Zyl commented on MNG-3647:
------------------------------------
All the hotspots across Plexus and Maven have been fixed in 3.x.
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2 & 3
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
> Fix For: 3.x
>
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Ittay Dror (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=140390#action_140390 ]
Ittay Dror commented on MNG-3647:
---------------------------------
I know about maven lifecycle. I'm just saying that having the mojo resources:resources called even when i have no resources wastes time.
It would be better if I could configure plugins in a parent pom and then easily enable/disable them in child poms. Then, even if 'resources:resources' is set in the default lifecycle, I can turn it off in a module that I know is not using it.
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Ittay Dror (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=140386#action_140386 ]
Ittay Dror commented on MNG-3647:
---------------------------------
See my example of abdera. It is dead simple: all simple jars. i run 'compile' and i skip tests. so i'd expect just the compile mojo to run. but maven will run every registered mojo.
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Ittay Dror (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=140338#action_140338 ]
Ittay Dror commented on MNG-3647:
---------------------------------
I forgot to mention that I ran maven in offline mode
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Milos Kleint (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=140388#action_140388 ]
Milos Kleint commented on MNG-3647:
-----------------------------------
read about maven lifecycle. http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
"mvn compile" will iterate phases up to the compile one and execute every mojo bound to the given phases. it either executes the mojos correctly or your project is wrongly setup.
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MNG-3647) Maven performance needs improvement
Posted by "Milos Kleint (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=140339#action_140339 ]
Milos Kleint commented on MNG-3647:
-----------------------------------
i've done my own profiling a while back (on svn 2.1-SNAPSHOT trunk )
http://www.mail-archive.com/dev@maven.apache.org/msg71597.html
actual execution of all plugins took only like 5 percent of the time. The rest was initial configuration overhead.
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MNG-3647) Maven performance needs improvement
Posted by "Brett Porter (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MNG-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brett Porter updated MNG-3647:
------------------------------
Fix Version/s: 2.0.x
it might be best to break this into specifically addressable issues
> Maven performance needs improvement
> -----------------------------------
>
> Key: MNG-3647
> URL: http://jira.codehaus.org/browse/MNG-3647
> Project: Maven 2
> Issue Type: Bug
> Affects Versions: 2.0.8
> Reporter: Ittay Dror
> Fix For: 2.0.x
>
>
> I have a multi-module project with 40 modules. Running 'install' with no compilation takes 30 seconds. Running 'compile' takes 18 seconds. This is very high IMHO. Buildr (written in Ruby - a scripted language) takes 1 second). In 'compile', adding time measurements to the mojo executions showed a cumulative time of 4.8 seconds. (Note that I ran maven several times, so all files are in the buffer cache)
> I've profiled the code to find obvious bottlenecks. Here is what I could easily fix:
> a. reading component configuration: in maven-uber jar there are 9 configurations that maven read and parsed 2394 times. I added a simple HashMap cache to return the already parsed configuration. Note that this suggest a lot of inefficiency in the maven code itself.
> b. ComponentValueSetter is used to set values in Mojos. It is created per field and always tries to find the field again. This showed high during profiling. I implemented a cache but I'm not sure whether this matters much
> c. in plexus, component discovery is done a lot of times - every time a jar is added to the container after it is started. this does a lot of xml parsing. I added code to disable component discovery temporarily and start it again. I call it from DefaultLifecycleExecutor.findExtensions at the start and end of the method. Also from DefaultPluginManager.ensurePluginContainerIsComplete.
> d. in DefaultRepositoryMetadataManager the function readMetadata always loads the metadata file and parses it. I have added a cache (although without paying attention to timestamps)
> Also, I ran java with the flags -client -dsa -da -Xnoclassgc -Xbatch -Xincgc. This shaved 3 seconds.
> All the above steps caused the time to drop to 8 seconds. Meaning 3 seconds due to JVM flags and 7 seconds of actions that could easily be avoided.
> There are other issues which are harder to tackle:
> 1. profiling shows that Xpp3DomBuilder is called a lot of times. Since it is called with a Reader/InputStream I can't easily implement a cache here.
> 2. jar files are always created and always installed. unlike the above actions, where the files were in the buffer cache, here actual IO occurs.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira