You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Sandy McPherson (JIRA)" <ji...@codehaus.org> on 2013/06/07 21:22:03 UTC
[jira] (MNG-5315) Artifact resolution sporadically fails in
parallel builds
[ https://jira.codehaus.org/browse/MNG-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=326367#comment-326367 ]
Sandy McPherson edited comment on MNG-5315 at 6/7/13 2:21 PM:
--------------------------------------------------------------
After a few false starts I found a plausible solution.
I Patched to only allow one thread at a time to do dependency resolution. I tried to do this with finer granularity (per artifact) but failed miserably due to the program structure. However the performance gain we have achieved in the builds we were targeting was one order of magnitude!
{code}
DefaultProjectDependenciesResolver
public DependencyResolutionResult resolve(
DependencyResolutionRequest request)
throws DependencyResolutionException {
long start = System.currentTimeMillis();
System.out.println("0 **** locking "+DefaultProjectDependenciesResolver.class);
synchronized (DefaultProjectDependenciesResolver.class) {
System.out.println((System.currentTimeMillis()-start)+" **** locked "+DefaultProjectDependenciesResolver.class);
RequestTrace trace = RequestTrace.newChild(null, request);
.
.
.
.
.
System.out.println((System.currentTimeMillis()-start)+" **** unlocked "+DefaultProjectDependenciesResolver.class);
return result;
}
{code}
Results are now
No failures so far. Massive speed improvements on larger builds with similar dependencies:
serial: 10-11m
-T 1C 1.5m
-T 2C 1.25m
-T 3C nearly 1m
was (Author: sandymcp):
After a few false starts I found a plausible solution.
I Patched to only allow one thread at a time to do dependency resolution. I tried to do this with finer granularity (per artifact) but failed miserably due to the program structure. However the performance gain we have achieved in the builds we were targeting was one order of magnitude!
DefaultProjectDependenciesResolver
public DependencyResolutionResult resolve(
DependencyResolutionRequest request)
throws DependencyResolutionException {
long start = System.currentTimeMillis();
System.out.println("0 **** locking "+DefaultProjectDependenciesResolver.class);
synchronized (DefaultProjectDependenciesResolver.class) {
System.out.println((System.currentTimeMillis()-start)+" **** locked "+DefaultProjectDependenciesResolver.class);
RequestTrace trace = RequestTrace.newChild(null, request);
.
.
.
.
.
System.out.println((System.currentTimeMillis()-start)+" **** unlocked "+DefaultProjectDependenciesResolver.class);
return result;
}
Results are now
No failures so far. Massive speed improvements on larger builds with similar dependencies:
serial: 10-11m
-T 1C 1.5m
-T 2C 1.25m
-T 3C nearly 1m
> Artifact resolution sporadically fails in parallel builds
> ---------------------------------------------------------
>
> Key: MNG-5315
> URL: https://jira.codehaus.org/browse/MNG-5315
> Project: Maven 2 & 3
> Issue Type: Bug
> Components: Artifacts and Repositories
> Affects Versions: 3.0.4
> Reporter: Ivan S. Dubrov
> Priority: Minor
>
> Artifact resolution can fail during the parallel build if it was downloaded during the same session.
> Scenario:
> 1) Delete the whole Maven local repository.
> 2) Run build "mvn compile -T1.5C"
> 3) Build fails (see log extracts below)
> 4) If I run build again, it succeeds.
> It seems like the problem is due to two thread trying to resolve same artifact concurrently. This problem never happens once I have all dependencies cached in the local repository.
> Extracts from the log output:
> {noformat}Downloading: http://nexus/content/repositories/thirdparty/com/googlecode/guava-osgi/guava-osgi/11.0.0/guava-osgi-11.0.0.jar 12444/13937 KB ...
> ...
> [DEBUG] Skipped remote update check for commons-cli:commons-cli:jar:1.2, already updated during this session.
> [DEBUG] Skipped remote update check for com.googlecode.guava-osgi:guava-osgi:jar:11.0.0, already updated during this session.
> [DEBUG] Skipped remote update check for org.slf4j:slf4j-api:jar:1.6.4, already updated during this session.
> [DEBUG] Skipped remote update check for org.slf4j:slf4j-jdk14:jar:1.6.4, already updated during this session.
> ...
> [ERROR] Failed to execute goal on project util: Could not resolve dependencies for project com.guidewire.pl:util:bundle:1.0-SNAPSHOT: The following artifacts could not be resolved: commons-cli:commons-cli:jar:1.2, com.googlecode.guava-osgi:guava-osgi:jar:11.0.0, org.slf4j:slf4j-api:jar:1.6.4, org.slf4j:slf4j-jdk14:jar:1.6.4: Failure to find commons-cli:commons-cli:jar:1.2 in http://nexus/content/repositories/thirdparty was cached in the local repository, resolution will not be reattempted until the update interval of gw.thirdparty has elapsed or updates are forced -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project util: Could not resolve dependencies for project com.guidewire.pl:util:bundle:1.0-SNAPSHOT: The following artifacts could not be resolved: commons-cli:commons-cli:jar:1.2, com.googlecode.guava-osgi:guava-osgi:jar:11.0.0, org.slf4j:slf4j-api:jar:1.6.4, org.slf4j:slf4j-jdk14:jar:1.6.4: Failure to find commons-cli:commons-cli:jar:1.2 in http://nexus/content/repositories/thirdparty was cached in the local repository, resolution will not be reattempted until the update interval of gw.thirdparty has elapsed or updates are forced
> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:210)
> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:117)
> at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:258)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:201)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:167)
> at org.apache.maven.lifecycle.internal.LifecycleThreadedBuilder$1.call(LifecycleThreadedBuilder.java:163)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project com.guidewire.pl:util:bundle:1.0-SNAPSHOT: The following artifacts could not be resolved: commons-cli:commons-cli:jar:1.2, com.googlecode.guava-osgi:guava-osgi:jar:11.0.0, org.slf4j:slf4j-api:jar:1.6.4, org.slf4j:slf4j-jdk14:jar:1.6.4: Failure to find commons-cli:commons-cli:jar:1.2 in http://nexus/content/repositories/thirdparty was cached in the local repository, resolution will not be reattempted until the update interval of gw.thirdparty has elapsed or updates are forced
> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:189)
> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:185)
> ... 16 more
> Caused by: org.sonatype.aether.resolution.DependencyResolutionException: The following artifacts could not be resolved: commons-cli:commons-cli:jar:1.2, com.googlecode.guava-osgi:guava-osgi:jar:11.0.0, org.slf4j:slf4j-api:jar:1.6.4, org.slf4j:slf4j-jdk14:jar:1.6.4: Failure to find commons-cli:commons-cli:jar:1.2 in http://nexus/content/repositories/thirdparty was cached in the local repository, resolution will not be reattempted until the update interval of gw.thirdparty has elapsed or updates are forced
> at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:375)
> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:183)
> ... 17 more
> Caused by: org.sonatype.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: commons-cli:commons-cli:jar:1.2, com.googlecode.guava-osgi:guava-osgi:jar:11.0.0, org.slf4j:slf4j-api:jar:1.6.4, org.slf4j:slf4j-jdk14:jar:1.6.4: Failure to find commons-cli:commons-cli:jar:1.2 in http://nexus/content/repositories/thirdparty was cached in the local repository, resolution will not be reattempted until the update interval of gw.thirdparty has elapsed or updates are forced
> at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:538)
> at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:216)
> at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:358)
> ... 18 more
> Caused by: org.sonatype.aether.transfer.ArtifactNotFoundException: Failure to find commons-cli:commons-cli:jar:1.2 in http://nexus/content/repositories/thirdparty was cached in the local repository, resolution will not be reattempted until the update interval of gw.thirdparty has elapsed or updates are forced
> at org.sonatype.aether.impl.internal.DefaultUpdateCheckManager.newException(DefaultUpdateCheckManager.java:230)
> at org.sonatype.aether.impl.internal.DefaultUpdateCheckManager.checkArtifact(DefaultUpdateCheckManager.java:176)
> at org.sonatype.aether.impl.internal.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:427)
> ... 20 more
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira