You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Jason Chaffee (JIRA)" <ji...@codehaus.org> on 2009/09/09 09:40:31 UTC
[jira] Created: (MDEP-230) Performance is really bad for large
artifacts
Performance is really bad for large artifacts
---------------------------------------------
Key: MDEP-230
URL: http://jira.codehaus.org/browse/MDEP-230
Project: Maven 2.x Dependency Plugin
Issue Type: Improvement
Components: unpack
Affects Versions: 2.1
Environment: linux
Reporter: Jason Chaffee
Assignee: Brian Fox
I have a 300mb tar.gz file that I need to unpack for one of my builds. I takes over 10 minutes to unpack with the unpack goal. If I do it on the cmd-line it takes less than 1 minute.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (MDEP-230) Performance is really bad for large artifacts
Posted by "Ahmed El-Madhoun (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/MDEP-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=317363#comment-317363 ]
Ahmed El-Madhoun commented on MDEP-230:
---------------------------------------
Is there anyway we can utilize the Java 7 features, add support for symlink discovery and creation?
> Performance is really bad for large artifacts
> ---------------------------------------------
>
> Key: MDEP-230
> URL: https://jira.codehaus.org/browse/MDEP-230
> Project: Maven 2.x Dependency Plugin
> Issue Type: Improvement
> Components: unpack
> Affects Versions: 2.1
> Environment: linux
> Reporter: Jason Chaffee
>
> I have a 300mb tar.gz file that I need to unpack for one of my builds. I takes over 10 minutes to unpack with the unpack goal. If I do it on the cmd-line it takes less than 1 minute.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MDEP-230) Performance is really bad for large
artifacts
Posted by "Mark Bowman (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MDEP-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=191138#action_191138 ]
Mark Bowman commented on MDEP-230:
----------------------------------
I am having similar problems. I have a tar.gz file which is 0.7Mb. It takes 3.5 minutes to unpack. The expansion to a temporary tar file is fast but the subsequent tar expansion is very slow. In fact the expansion seems to happen very quickly but the plugin than sits doing nothing (according to mvn -X) for several minutes. It doesn't even complain if you remove the tar file it reports to be expanding.
> Performance is really bad for large artifacts
> ---------------------------------------------
>
> Key: MDEP-230
> URL: http://jira.codehaus.org/browse/MDEP-230
> Project: Maven 2.x Dependency Plugin
> Issue Type: Improvement
> Components: unpack
> Affects Versions: 2.1
> Environment: linux
> Reporter: Jason Chaffee
> Assignee: Brian Fox
>
> I have a 300mb tar.gz file that I need to unpack for one of my builds. I takes over 10 minutes to unpack with the unpack goal. If I do it on the cmd-line it takes less than 1 minute.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (MDEP-230) Performance is really bad for large artifacts
Posted by "Olivier Lamy (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/MDEP-230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olivier Lamy closed MDEP-230.
-----------------------------
Resolution: Won't Fix
Assignee: Olivier Lamy
already fixed with recent version.
> Performance is really bad for large artifacts
> ---------------------------------------------
>
> Key: MDEP-230
> URL: https://jira.codehaus.org/browse/MDEP-230
> Project: Maven 2.x Dependency Plugin
> Issue Type: Improvement
> Components: unpack
> Affects Versions: 2.1
> Environment: linux
> Reporter: Jason Chaffee
> Assignee: Olivier Lamy
>
> I have a 300mb tar.gz file that I need to unpack for one of my builds. I takes over 10 minutes to unpack with the unpack goal. If I do it on the cmd-line it takes less than 1 minute.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (MDEP-230) Performance is really bad for large artifacts
Posted by "Ivan (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/MDEP-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=316942#comment-316942 ]
Ivan commented on MDEP-230:
---------------------------
My guess is that the implementation of "chmod" is 'inefficient', it seems to fallback to a Runtime.exec("chmod...") for each file.
(If not using a Java7 jvm)
Here is a jstack:
"main" prio=10 tid=0x000000004106b000 nid=0x342c in Object.wait() [0x00000000408fd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab5b31c890> (a java.lang.UNIXProcess$Gate)
at java.lang.Object.wait(Object.java:485)
at java.lang.UNIXProcess$Gate.waitForExit(UNIXProcess.java:64)
- locked <0x00002aab5b31c890> (a java.lang.UNIXProcess$Gate)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:145)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
at java.lang.Runtime.exec(Runtime.java:593)
at org.codehaus.plexus.util.cli.Commandline.execute(Commandline.java:651)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLineAsCallable(CommandLineUtils.java:140)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine(CommandLineUtils.java:111)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine(CommandLineUtils.java:79)
at org.codehaus.plexus.archiver.util.ArchiveEntryUtils.chmod(ArchiveEntryUtils.java:92)
at org.codehaus.plexus.archiver.zip.AbstractZipUnArchiver.extractFile(AbstractZipUnArchiver.java:238)
at org.codehaus.plexus.archiver.zip.AbstractZipUnArchiver.extractFileIfIncluded(AbstractZipUnArchiver.java:185)
at org.codehaus.plexus.archiver.zip.AbstractZipUnArchiver.execute(AbstractZipUnArchiver.java:149)
at org.codehaus.plexus.archiver.AbstractUnArchiver.extract(AbstractUnArchiver.java:119)
at org.apache.maven.plugin.dependency.AbstractDependencyMojo.unpack(AbstractDependencyMojo.java:266)
at org.apache.maven.plugin.dependency.UnpackDependenciesMojo.execute(UnpackDependenciesMojo.java:91)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> Performance is really bad for large artifacts
> ---------------------------------------------
>
> Key: MDEP-230
> URL: https://jira.codehaus.org/browse/MDEP-230
> Project: Maven 2.x Dependency Plugin
> Issue Type: Improvement
> Components: unpack
> Affects Versions: 2.1
> Environment: linux
> Reporter: Jason Chaffee
>
> I have a 300mb tar.gz file that I need to unpack for one of my builds. I takes over 10 minutes to unpack with the unpack goal. If I do it on the cmd-line it takes less than 1 minute.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (MDEP-230) Performance is really bad for large artifacts
Posted by "Olivier Lamy (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/MDEP-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=317366#comment-317366 ]
Olivier Lamy commented on MDEP-230:
-----------------------------------
please upgrade your version to 2.5.1 and have a look at http://maven.apache.org/plugins/maven-dependency-plugin/unpack-mojo.html#useJvmChmod
which is true per default.
> Performance is really bad for large artifacts
> ---------------------------------------------
>
> Key: MDEP-230
> URL: https://jira.codehaus.org/browse/MDEP-230
> Project: Maven 2.x Dependency Plugin
> Issue Type: Improvement
> Components: unpack
> Affects Versions: 2.1
> Environment: linux
> Reporter: Jason Chaffee
>
> I have a 300mb tar.gz file that I need to unpack for one of my builds. I takes over 10 minutes to unpack with the unpack goal. If I do it on the cmd-line it takes less than 1 minute.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MDEP-230) Performance is really bad for large
artifacts
Posted by "Mark Bowman (JIRA)" <ji...@codehaus.org>.
[ http://jira.codehaus.org/browse/MDEP-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=191279#action_191279 ]
Mark Bowman commented on MDEP-230:
----------------------------------
I've been having a quick look into this by checking out the
maven-dependency-plugin source code and adding debug:-
http://svn.apache.org/repos/asf/maven/plugins/tags/maven-dependency-plugin-2.1
It appears to be slowing down in the extract method within the TarUnArchiver
class. This class is part of the plexus-archiver-1.0-alpha-9 package.
Checking this package out and inserting debug:-
http://svn.codehaus.org/plexus/plexus-components/tags/plexus-archiver-1.0-alpha-9
shows that the TarUnarchiver class is extracting individual files and taking
~50ms per file.
> Performance is really bad for large artifacts
> ---------------------------------------------
>
> Key: MDEP-230
> URL: http://jira.codehaus.org/browse/MDEP-230
> Project: Maven 2.x Dependency Plugin
> Issue Type: Improvement
> Components: unpack
> Affects Versions: 2.1
> Environment: linux
> Reporter: Jason Chaffee
> Assignee: Brian Fox
>
> I have a 300mb tar.gz file that I need to unpack for one of my builds. I takes over 10 minutes to unpack with the unpack goal. If I do it on the cmd-line it takes less than 1 minute.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (MDEP-230) Performance is really bad for large artifacts
Posted by "Ivan (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/MDEP-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=316942#comment-316942 ]
Ivan edited comment on MDEP-230 at 1/8/13 3:46 AM:
---------------------------------------------------
My guess is that the implementation of org.codehaus.plexus.archiver.util.ArchiveEntryUtils.chmod is cpu consuming.
Indeed it seems to fallback to a Runtime.exec("chmod...") for each file.
(If not using a Java7 jvm)
Here is a jstack:
"main" prio=10 tid=0x000000004106b000 nid=0x342c in Object.wait() [0x00000000408fd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab5b31c890> (a java.lang.UNIXProcess$Gate)
at java.lang.Object.wait(Object.java:485)
at java.lang.UNIXProcess$Gate.waitForExit(UNIXProcess.java:64)
- locked <0x00002aab5b31c890> (a java.lang.UNIXProcess$Gate)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:145)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
at java.lang.Runtime.exec(Runtime.java:593)
at org.codehaus.plexus.util.cli.Commandline.execute(Commandline.java:651)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLineAsCallable(CommandLineUtils.java:140)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine(CommandLineUtils.java:111)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine(CommandLineUtils.java:79)
at org.codehaus.plexus.archiver.util.ArchiveEntryUtils.chmod(ArchiveEntryUtils.java:92)
at org.codehaus.plexus.archiver.zip.AbstractZipUnArchiver.extractFile(AbstractZipUnArchiver.java:238)
at org.codehaus.plexus.archiver.zip.AbstractZipUnArchiver.extractFileIfIncluded(AbstractZipUnArchiver.java:185)
at org.codehaus.plexus.archiver.zip.AbstractZipUnArchiver.execute(AbstractZipUnArchiver.java:149)
at org.codehaus.plexus.archiver.AbstractUnArchiver.extract(AbstractUnArchiver.java:119)
at org.apache.maven.plugin.dependency.AbstractDependencyMojo.unpack(AbstractDependencyMojo.java:266)
at org.apache.maven.plugin.dependency.UnpackDependenciesMojo.execute(UnpackDependenciesMojo.java:91)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
was (Author: ebann):
My guess is that the implementation of "chmod" is 'inefficient', it seems to fallback to a Runtime.exec("chmod...") for each file.
(If not using a Java7 jvm)
Here is a jstack:
"main" prio=10 tid=0x000000004106b000 nid=0x342c in Object.wait() [0x00000000408fd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aab5b31c890> (a java.lang.UNIXProcess$Gate)
at java.lang.Object.wait(Object.java:485)
at java.lang.UNIXProcess$Gate.waitForExit(UNIXProcess.java:64)
- locked <0x00002aab5b31c890> (a java.lang.UNIXProcess$Gate)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:145)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
at java.lang.Runtime.exec(Runtime.java:593)
at org.codehaus.plexus.util.cli.Commandline.execute(Commandline.java:651)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLineAsCallable(CommandLineUtils.java:140)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine(CommandLineUtils.java:111)
at org.codehaus.plexus.util.cli.CommandLineUtils.executeCommandLine(CommandLineUtils.java:79)
at org.codehaus.plexus.archiver.util.ArchiveEntryUtils.chmod(ArchiveEntryUtils.java:92)
at org.codehaus.plexus.archiver.zip.AbstractZipUnArchiver.extractFile(AbstractZipUnArchiver.java:238)
at org.codehaus.plexus.archiver.zip.AbstractZipUnArchiver.extractFileIfIncluded(AbstractZipUnArchiver.java:185)
at org.codehaus.plexus.archiver.zip.AbstractZipUnArchiver.execute(AbstractZipUnArchiver.java:149)
at org.codehaus.plexus.archiver.AbstractUnArchiver.extract(AbstractUnArchiver.java:119)
at org.apache.maven.plugin.dependency.AbstractDependencyMojo.unpack(AbstractDependencyMojo.java:266)
at org.apache.maven.plugin.dependency.UnpackDependenciesMojo.execute(UnpackDependenciesMojo.java:91)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> Performance is really bad for large artifacts
> ---------------------------------------------
>
> Key: MDEP-230
> URL: https://jira.codehaus.org/browse/MDEP-230
> Project: Maven 2.x Dependency Plugin
> Issue Type: Improvement
> Components: unpack
> Affects Versions: 2.1
> Environment: linux
> Reporter: Jason Chaffee
>
> I have a 300mb tar.gz file that I need to unpack for one of my builds. I takes over 10 minutes to unpack with the unpack goal. If I do it on the cmd-line it takes less than 1 minute.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira