You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Laurence Bordowitz <lb...@yahoo-inc.com.INVALID> on 2015/04/10 20:35:19 UTC
Parallel builds and dependency resolution
I'm having a problem where building a large multi-module project in parallel causes the dependency resolution to fail: org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project [PROJECT]: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
Normally, the answer to this would be, "you didn't set up your repository correctly, check your company's mirrors, etc." However, our builds usually succeed, and all the dependencies resolve. In fact, this only happens in about 10% of the builds, and with different jars (although often the same set of jars). Earlier in the same build where the log line above comes from, we get this message:
<b>17:53:20</b> [INFO] Downloading: http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar
<b>17:53:20</b> [INFO] Changes detected - recompiling the module!
<b>17:53:20</b> [INFO] Compiling 7 source files to [/location/of/different/project]
<b>17:53:20</b> [INFO]
<b>17:53:20</b> [INFO] --- maven-clean-plugin:2.6:clean (default-clean) @ [Yet another project] ---
<b>17:53:20</b> [INFO] Downloaded: http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar (93 KB at 11504.3 KB/sec)
Moreover, some dependencies are being downloaded twice. This has led me to the conclusion that there's a race condition happening with dependencies. Two modules, in parallel, detect that a resource is missing. They both download it. Sometime between the opening and use of a dependency resource in one module, another download from a parallel-built module finishes downloading and rewrites the file. Suddenly, the resource is missing!
Is this right? Has anybody else had this problem? Does anybody know of any workaround? I tried mvn dependency:go-offline, but that requires dependencies that aren't built yet, and its group-id exclusion doesn't work when it's not a plugin.
Here's the stack trace:
<b>17:53:20</b> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project [PROJECT]: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:220)
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:188)
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:184)
<b>17:53:20</b> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
<b>17:53:20</b> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
<b>17:53:20</b> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
<b>17:53:20</b> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
<b>17:53:20</b> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
<b>17:53:20</b> at java.lang.Thread.run(Thread.java:745)
<b>17:53:20</b> Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
<b>17:53:20</b> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:198)
<b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
<b>17:53:20</b> ... 14 more
<b>17:53:20</b> Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
<b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)
<b>17:53:20</b> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:192)
<b>17:53:20</b> ... 15 more
<b>17:53:20</b> Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
<b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:459)
<b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
<b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)
<b>17:53:20</b> ... 16 more
<b>17:53:20</b> Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
<b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:449)
<b>17:53:20</b> ... 18 more
-- Larry Bordowitz
Re: Parallel builds and dependency resolution
Posted by Karl Heinz Marbaise <kh...@gmx.de>.
Hi Laurence,
On 4/10/15 9:29 PM, Laurence Bordowitz wrote:
> >Remove every content from your local repository ($HOME/.m2/repository
> >simply empty)...and just do:
> >
> >mvn package
> >
> >does this work without any problem..?
>
> The command we use for building is
>
> mvn -P[profile] clean install -T 1C
No please really check with an empty repository and simply:
mvn clean package
No -T option and do not use "install" !!!
I recommend to check your build first via the above...
BTW: How many modules do you have? Do you know the number of lines of
code (may be counted by Sonaqube) ?
>
> And the profile is only used at the very end of the build for some
> proprietary packaging thing.
>
> >Which Maven version do you use?
>
> Apache Maven 3.2.1, using Java version 1.8.0.31
with JDK 1.8.0.40 ...there have been fixed some bugs which had causes
some issues as far as i know....
>
> >Which Maven Plugins do you use in which versions? Where does this
> >problem happen?
> >
> >From which of your plugins is this message emitted ?
It is really important to know which plugin versions you use....cause
there had been some problems in parallelization....
Simplest is if you create a log file of the build run and do the following:
cat mvn.log | egrep -a "\[INFO\] --- " | cut -d" " -f3 | sort -u
or you can use the log file which is produced for example from Jenkins
on console. This will give a list of plugins with their appropriate
versions without any internal details only a list like this:
appassembler-maven-plugin:1.9:assemble
maven-assembly-plugin:2.5.3:single
maven-clean-plugin:2.6.1:clean
maven-compiler-plugin:3.2:compile
maven-compiler-plugin:3.2:testCompile
maven-ear-plugin:2.10:ear
maven-ear-plugin:2.10:generate-application-xml
maven-ejb-plugin:2.5:ejb
maven-enforcer-plugin:1.4:enforce
maven-jar-plugin:2.5:jar
maven-resources-plugin:2.7:resources
maven-resources-plugin:2.7:testResources
maven-shade-plugin:2.3:shade
maven-surefire-plugin:2.18.1:test
maven-war-plugin:2.6:war
Things like this had been there:
https://issues.apache.org/jira/browse/MEAR-201?jql=project%20%3D%20MEAR%20AND%20fixVersion%20%3D%202.10
The problem does not really look like a typical concurrent issue...
>
> Not sure... it just says:
>
> [INFO] Building [project_name] 1.1-SNAPSHOT
>
> I guess that's the maven compiler plugin?
>
> >Does this happen every time at the same are or more on coincidence?
>
> Every time that it does happen, it always happens around the same
> compile duration, around five or six minutes into the usually 30 minute
> build. Haven't kept track of whether it happens during the same absolute
> time, but that might be a good thing to track.
you should check this...
>
> >BTW: I assume you are using a repository manager ? Which one? In which
> >version?
>
> It's an internally developed proxy system; if the dependency can't be
> resolved internally, it goes to a cache.
Why not using a repository manager like Nexus, Artifactory or Archiva ?
>
> -- Larry Bordowitz
>
>
>
> On Friday, April 10, 2015 12:00 PM, Karl Heinz Marbaise
> <kh...@gmx.de> wrote:
>
>
> Hi Laurence,
>
> before you are searching phantoms i would suggest to first check if you
> can build without any issue by the following reprequisites on command line:
>
> Remove every content from your local repository ($HOME/.m2/repository
> simply empty)...and just do:
>
> mvn package
>
> does this work without any problem..?
>
>
> Furthermore could you give more information like the following:
>
> Which Maven version do you use?
>
> Which Maven Plugins do you use in which versions? Where does this
> problem happen?
>
> From which of your plugins is this message emitted ?
>
> Does this happen every time at the same are or more on coincidence?
>
> BTW: I assume you are using a repository manager ? Which one? In which
> version?
>
>
> Kind regards
> Karl Heinz Marbaise
>
> On 4/10/15 8:35 PM, Laurence Bordowitz wrote:
> > I'm having a problem where building a large multi-module project in
> parallel causes the dependency resolution to fail:
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> execute goal on project [PROJECT]: Could not resolve dependencies for
> project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could
> not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> >
> > Normally, the answer to this would be, "you didn't set up your
> repository correctly, check your company's mirrors, etc." However, our
> builds usually succeed, and all the dependencies resolve. In fact, this
> only happens in about 10% of the builds, and with different jars
> (although often the same set of jars). Earlier in the same build where
> the log line above comes from, we get this message:
> > <b>17:53:20</b> [INFO] Downloading:
> http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar
> > <b>17:53:20</b> [INFO] Changes detected - recompiling the module!
> > <b>17:53:20</b> [INFO] Compiling 7 source files to
> [/location/of/different/project]
> > <b>17:53:20</b> [INFO]
> > <b>17:53:20</b> [INFO] --- maven-clean-plugin:2.6:clean
> (default-clean) @ [Yet another project] ---
> > <b>17:53:20</b> [INFO] Downloaded:
> http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar
> (93 KB at 11504.3 KB/sec)
> >
> > Moreover, some dependencies are being downloaded twice. This has led
> me to the conclusion that there's a race condition happening with
> dependencies. Two modules, in parallel, detect that a resource is
> missing. They both download it. Sometime between the opening and use of
> a dependency resource in one module, another download from a
> parallel-built module finishes downloading and rewrites the file.
> Suddenly, the resource is missing!
> > Is this right? Has anybody else had this problem? Does anybody know
> of any workaround? I tried mvn dependency:go-offline, but that requires
> dependencies that aren't built yet, and its group-id exclusion doesn't
> work when it's not a plugin.
> >
> > Here's the stack trace:
> > <b>17:53:20</b>
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to
> execute goal on project [PROJECT]: Could not resolve dependencies for
> project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could
> not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:220)
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:188)
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:184)
> > <b>17:53:20</b> at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > <b>17:53:20</b> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > <b>17:53:20</b> at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > <b>17:53:20</b> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > <b>17:53:20</b> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > <b>17:53:20</b> at java.lang.Thread.run(Thread.java:745)
> > <b>17:53:20</b> Caused by:
> org.apache.maven.project.DependencyResolutionException: Could not
> resolve dependencies for project
> [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find
> artifact com.jayway.jsonpath:json-path:jar:0.9.1
> > <b>17:53:20</b> at
> org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:198)
> > <b>17:53:20</b> at
> org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
> > <b>17:53:20</b> ... 14 more
> > <b>17:53:20</b> Caused by:
> org.eclipse.aether.resolution.DependencyResolutionException: Could not
> find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> > <b>17:53:20</b> at
> org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)
> > <b>17:53:20</b> at
> org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:192)
> > <b>17:53:20</b> ... 15 more
> > <b>17:53:20</b> Caused by:
> org.eclipse.aether.resolution.ArtifactResolutionException: Could not
> find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> > <b>17:53:20</b> at
> org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:459)
> > <b>17:53:20</b> at
> org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
> > <b>17:53:20</b> at
> org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)
> > <b>17:53:20</b> ... 16 more
> > <b>17:53:20</b> Caused by:
> org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find
> artifact com.jayway.jsonpath:json-path:jar:0.9.1
> > <b>17:53:20</b> at
> org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:449)
> > <b>17:53:20</b> ... 18 more
> >
> > -- Larry Bordowitz
>
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Parallel builds and dependency resolution
Posted by Jason van Zyl <ja...@takari.io>.
You may want to take a look at the following:
https://github.com/takari/core-extensions-example
On Apr 10, 2015, at 3:29 PM, Laurence Bordowitz <lb...@yahoo-inc.com.INVALID> wrote:
>> Remove every content from your local repository ($HOME/.m2/repository
>> simply empty)...and just do:
>>
>> mvn package
>>
>> does this work without any problem..?
> The command we use for building is
>
> mvn -P[profile] clean install -T 1C
> And the profile is only used at the very end of the build for some proprietary packaging thing.
>> Which Maven version do you use?
> Apache Maven 3.2.1, using Java version 1.8.0.31
>
>> Which Maven Plugins do you use in which versions? Where does this
>> problem happen?
>>
>> From which of your plugins is this message emitted ?
> Not sure... it just says:
> [INFO] Building [project_name] 1.1-SNAPSHOT
> I guess that's the maven compiler plugin?
>
>> Does this happen every time at the same are or more on coincidence?
> Every time that it does happen, it always happens around the same compile duration, around five or six minutes into the usually 30 minute build. Haven't kept track of whether it happens during the same absolute time, but that might be a good thing to track.
>
>> BTW: I assume you are using a repository manager ? Which one? In which
>> version?
> It's an internally developed proxy system; if the dependency can't be resolved internally, it goes to a cache.
>
> -- Larry Bordowitz
>
>
>
> On Friday, April 10, 2015 12:00 PM, Karl Heinz Marbaise <kh...@gmx.de> wrote:
>
>
> Hi Laurence,
>
> before you are searching phantoms i would suggest to first check if you
> can build without any issue by the following reprequisites on command line:
>
> Remove every content from your local repository ($HOME/.m2/repository
> simply empty)...and just do:
>
> mvn package
>
> does this work without any problem..?
>
>
> Furthermore could you give more information like the following:
>
> Which Maven version do you use?
>
> Which Maven Plugins do you use in which versions? Where does this
> problem happen?
>
> From which of your plugins is this message emitted ?
>
> Does this happen every time at the same are or more on coincidence?
>
> BTW: I assume you are using a repository manager ? Which one? In which
> version?
>
>
> Kind regards
> Karl Heinz Marbaise
>
> On 4/10/15 8:35 PM, Laurence Bordowitz wrote:
>> I'm having a problem where building a large multi-module project in parallel causes the dependency resolution to fail: org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project [PROJECT]: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
>>
>> Normally, the answer to this would be, "you didn't set up your repository correctly, check your company's mirrors, etc." However, our builds usually succeed, and all the dependencies resolve. In fact, this only happens in about 10% of the builds, and with different jars (although often the same set of jars). Earlier in the same build where the log line above comes from, we get this message:
>> <b>17:53:20</b> [INFO] Downloading: http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar
>> <b>17:53:20</b> [INFO] Changes detected - recompiling the module!
>> <b>17:53:20</b> [INFO] Compiling 7 source files to [/location/of/different/project]
>> <b>17:53:20</b> [INFO]
>> <b>17:53:20</b> [INFO] --- maven-clean-plugin:2.6:clean (default-clean) @ [Yet another project] ---
>> <b>17:53:20</b> [INFO] Downloaded: http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar (93 KB at 11504.3 KB/sec)
>>
>> Moreover, some dependencies are being downloaded twice. This has led me to the conclusion that there's a race condition happening with dependencies. Two modules, in parallel, detect that a resource is missing. They both download it. Sometime between the opening and use of a dependency resource in one module, another download from a parallel-built module finishes downloading and rewrites the file. Suddenly, the resource is missing!
>> Is this right? Has anybody else had this problem? Does anybody know of any workaround? I tried mvn dependency:go-offline, but that requires dependencies that aren't built yet, and its group-id exclusion doesn't work when it's not a plugin.
>>
>> Here's the stack trace:
>> <b>17:53:20</b> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project [PROJECT]: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:220)
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:188)
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:184)
>> <b>17:53:20</b> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> <b>17:53:20</b> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> <b>17:53:20</b> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> <b>17:53:20</b> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> <b>17:53:20</b> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> <b>17:53:20</b> at java.lang.Thread.run(Thread.java:745)
>> <b>17:53:20</b> Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
>> <b>17:53:20</b> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:198)
>> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
>> <b>17:53:20</b> ... 14 more
>> <b>17:53:20</b> Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
>> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)
>> <b>17:53:20</b> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:192)
>> <b>17:53:20</b> ... 15 more
>> <b>17:53:20</b> Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
>> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:459)
>> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
>> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)
>> <b>17:53:20</b> ... 16 more
>> <b>17:53:20</b> Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
>> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:449)
>> <b>17:53:20</b> ... 18 more
>>
>> -- Larry Bordowitz
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
>
Thanks,
Jason
----------------------------------------------------------
Jason van Zyl
Founder, Takari and Apache Maven
http://twitter.com/jvanzyl
http://twitter.com/takari_io
---------------------------------------------------------
We all have problems. How we deal with them is a measure of our worth.
-- Unknown
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Parallel builds and dependency resolution
Posted by Laurence Bordowitz <lb...@yahoo-inc.com.INVALID>.
>Remove every content from your local repository ($HOME/.m2/repository
>simply empty)...and just do:
>
>mvn package
>
>does this work without any problem..?
The command we use for building is
mvn -P[profile] clean install -T 1C
And the profile is only used at the very end of the build for some proprietary packaging thing.
>Which Maven version do you use?
Apache Maven 3.2.1, using Java version 1.8.0.31
>Which Maven Plugins do you use in which versions? Where does this
>problem happen?
>
>From which of your plugins is this message emitted ?
Not sure... it just says:
[INFO] Building [project_name] 1.1-SNAPSHOT
I guess that's the maven compiler plugin?
>Does this happen every time at the same are or more on coincidence?
Every time that it does happen, it always happens around the same compile duration, around five or six minutes into the usually 30 minute build. Haven't kept track of whether it happens during the same absolute time, but that might be a good thing to track.
>BTW: I assume you are using a repository manager ? Which one? In which
>version?
It's an internally developed proxy system; if the dependency can't be resolved internally, it goes to a cache.
-- Larry Bordowitz
On Friday, April 10, 2015 12:00 PM, Karl Heinz Marbaise <kh...@gmx.de> wrote:
Hi Laurence,
before you are searching phantoms i would suggest to first check if you
can build without any issue by the following reprequisites on command line:
Remove every content from your local repository ($HOME/.m2/repository
simply empty)...and just do:
mvn package
does this work without any problem..?
Furthermore could you give more information like the following:
Which Maven version do you use?
Which Maven Plugins do you use in which versions? Where does this
problem happen?
From which of your plugins is this message emitted ?
Does this happen every time at the same are or more on coincidence?
BTW: I assume you are using a repository manager ? Which one? In which
version?
Kind regards
Karl Heinz Marbaise
On 4/10/15 8:35 PM, Laurence Bordowitz wrote:
> I'm having a problem where building a large multi-module project in parallel causes the dependency resolution to fail: org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project [PROJECT]: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
>
> Normally, the answer to this would be, "you didn't set up your repository correctly, check your company's mirrors, etc." However, our builds usually succeed, and all the dependencies resolve. In fact, this only happens in about 10% of the builds, and with different jars (although often the same set of jars). Earlier in the same build where the log line above comes from, we get this message:
> <b>17:53:20</b> [INFO] Downloading: http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar
> <b>17:53:20</b> [INFO] Changes detected - recompiling the module!
> <b>17:53:20</b> [INFO] Compiling 7 source files to [/location/of/different/project]
> <b>17:53:20</b> [INFO]
> <b>17:53:20</b> [INFO] --- maven-clean-plugin:2.6:clean (default-clean) @ [Yet another project] ---
> <b>17:53:20</b> [INFO] Downloaded: http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar (93 KB at 11504.3 KB/sec)
>
> Moreover, some dependencies are being downloaded twice. This has led me to the conclusion that there's a race condition happening with dependencies. Two modules, in parallel, detect that a resource is missing. They both download it. Sometime between the opening and use of a dependency resource in one module, another download from a parallel-built module finishes downloading and rewrites the file. Suddenly, the resource is missing!
> Is this right? Has anybody else had this problem? Does anybody know of any workaround? I tried mvn dependency:go-offline, but that requires dependencies that aren't built yet, and its group-id exclusion doesn't work when it's not a plugin.
>
> Here's the stack trace:
> <b>17:53:20</b> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project [PROJECT]: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:220)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:188)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:184)
> <b>17:53:20</b> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> <b>17:53:20</b> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> <b>17:53:20</b> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> <b>17:53:20</b> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> <b>17:53:20</b> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> <b>17:53:20</b> at java.lang.Thread.run(Thread.java:745)
> <b>17:53:20</b> Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:198)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
> <b>17:53:20</b> ... 14 more
> <b>17:53:20</b> Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)
> <b>17:53:20</b> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:192)
> <b>17:53:20</b> ... 15 more
> <b>17:53:20</b> Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:459)
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)
> <b>17:53:20</b> ... 16 more
> <b>17:53:20</b> Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:449)
> <b>17:53:20</b> ... 18 more
>
> -- Larry Bordowitz
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Parallel builds and dependency resolution
Posted by Karl Heinz Marbaise <kh...@gmx.de>.
Hi Laurence,
before you are searching phantoms i would suggest to first check if you
can build without any issue by the following reprequisites on command line:
Remove every content from your local repository ($HOME/.m2/repository
simply empty)...and just do:
mvn package
does this work without any problem..?
Furthermore could you give more information like the following:
Which Maven version do you use?
Which Maven Plugins do you use in which versions? Where does this
problem happen?
From which of your plugins is this message emitted ?
Does this happen every time at the same are or more on coincidence?
BTW: I assume you are using a repository manager ? Which one? In which
version?
Kind regards
Karl Heinz Marbaise
On 4/10/15 8:35 PM, Laurence Bordowitz wrote:
> I'm having a problem where building a large multi-module project in parallel causes the dependency resolution to fail: org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project [PROJECT]: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
>
> Normally, the answer to this would be, "you didn't set up your repository correctly, check your company's mirrors, etc." However, our builds usually succeed, and all the dependencies resolve. In fact, this only happens in about 10% of the builds, and with different jars (although often the same set of jars). Earlier in the same build where the log line above comes from, we get this message:
> <b>17:53:20</b> [INFO] Downloading: http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar
> <b>17:53:20</b> [INFO] Changes detected - recompiling the module!
> <b>17:53:20</b> [INFO] Compiling 7 source files to [/location/of/different/project]
> <b>17:53:20</b> [INFO]
> <b>17:53:20</b> [INFO] --- maven-clean-plugin:2.6:clean (default-clean) @ [Yet another project] ---
> <b>17:53:20</b> [INFO] Downloaded: http://[MAVEN-REPO]/proximity/repository/public/com/jayway/jsonpath/json-path/0.9.1/json-path-0.9.1.jar (93 KB at 11504.3 KB/sec)
>
> Moreover, some dependencies are being downloaded twice. This has led me to the conclusion that there's a race condition happening with dependencies. Two modules, in parallel, detect that a resource is missing. They both download it. Sometime between the opening and use of a dependency resource in one module, another download from a parallel-built module finishes downloading and rewrites the file. Suddenly, the resource is missing!
> Is this right? Has anybody else had this problem? Does anybody know of any workaround? I tried mvn dependency:go-offline, but that requires dependencies that aren't built yet, and its group-id exclusion doesn't work when it's not a plugin.
>
> Here's the stack trace:
> <b>17:53:20</b> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project [PROJECT]: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:220)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:108)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:188)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:184)
> <b>17:53:20</b> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> <b>17:53:20</b> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> <b>17:53:20</b> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> <b>17:53:20</b> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> <b>17:53:20</b> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> <b>17:53:20</b> at java.lang.Thread.run(Thread.java:745)
> <b>17:53:20</b> Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project [company].[groupid].[PROJECT]:[PROJECT]:jar:1.1-SNAPSHOT: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:198)
> <b>17:53:20</b> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
> <b>17:53:20</b> ... 14 more
> <b>17:53:20</b> Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:384)
> <b>17:53:20</b> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:192)
> <b>17:53:20</b> ... 15 more
> <b>17:53:20</b> Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:459)
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:262)
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:367)
> <b>17:53:20</b> ... 16 more
> <b>17:53:20</b> Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact com.jayway.jsonpath:json-path:jar:0.9.1
> <b>17:53:20</b> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:449)
> <b>17:53:20</b> ... 18 more
>
> -- Larry Bordowitz
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org