You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "James B (JIRA)" <ji...@apache.org> on 2018/11/13 08:23:00 UTC
[jira] [Commented] (MDEP-518) Dependency resolution fails in
threaded builds
[ https://issues.apache.org/jira/browse/MDEP-518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684863#comment-16684863 ]
James B commented on MDEP-518:
------------------------------
I am also experiencing this issue. I'm using maven 3.5.0 but see no need to try on latest, as the previous comment shows 3.5.4 still broken.
I have a very large multi-module project with hundreds of modules and a complex reactor graph, thus lending itself very well to parallel building. With single-process build it works fine, with -T1.5C it fails fairly early on with something like:
{code:java}
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on project XXX: Unable to generate classpath: org.apache.maven.artifact.resolver.ArtifactResolutionException: Unable to get dependency information for org.apache.maven.surefire:surefire-junit-platform:jar:2.22.1: Failed to retrieve POM for org.apache.maven.surefire:surefire-junit-platform:jar:2.22.1: Could not transfer artifact org.apache.maven.surefire:surefire-junit-platform:pom:2.22.1 from/to REPO (https://repo.lan/artifactory/maven/maven2): /build/workspace/JOB/.repository/org/apache/maven/surefire/surefire-junit-platform/2.22.1/surefire-junit-platform-2.22.1.pom.part (No such file or directory)
[ERROR] org.apache.maven.surefire:surefire-junit-platform:jar:2.22.1
{code}
Note again that the file not being found is the ".part".
At first glance this appears to happen when two threads simultaneously download the same artifact. As surefire itself is failing, maybe this is specific to plugin dependency resolution? The original reported exception indicates the problem also happens on standard artifacts.
Appears related to a recently revived issue, with a new Pull Request: [MNG-2802|https://issues.apache.org/jira/browse/MNG-2802?focusedCommentId=16584418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16584418]
> Dependency resolution fails in threaded builds
> ----------------------------------------------
>
> Key: MDEP-518
> URL: https://issues.apache.org/jira/browse/MDEP-518
> Project: Maven Dependency Plugin
> Issue Type: Bug
> Components: resolve, resolve-plugins
> Affects Versions: 2.10
> Environment: Linux
> Reporter: Troy Telford
> Priority: Major
>
> This issue sounds similar to MDEP-442, but it may be unrelated.
> I suspect the same problem may occur in both {{dependency:resolve}} and {{dependency:resolve-plugins}}.
> The same error occurs when I use {{mvn install}}; but {{dependency:resolve}} is a more tightly-focused problem report.
> I have a large multi-module project, with more than 40 modules, and a large-ish build node with 24 cores. {{mvn -T1C dependency:resolve dependency:resolve-plugins}} will typically (94/100 attempts) fail to download all of the dependencies required, with an error similar to:
> {noformat}
> Failed to read artifact descriptor for com.foo:machine:jar:1.2.3-SNAPSHOT: Could not transfer artifact com.foo.somegroup:dependency_4:jar:3.0.0 from/to snapshot-repository (https://nexus.foo.com/content/repositories/releases): /home/userid/.m2/repository/com/foo/somegroup/dependency_4/3.0.0/dependency_4-3.0.0.pom.part (No such file or directory) -> [Help 1]
> -> com.foo.somegroup:dependency_4:jar:3.0.0
> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:220)
> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:127)
> at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved(MojoExecutor.java:257)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:200)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:188)
> at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:184)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project com.foo:machine:jar:1.2.3-SNAPSHOT: Failed to collect dependencies at com.foo:dependency_1:jar:1.2.3-SNAPSHOT -> com.foo:dependency_2:jar:1.2.3-SNAPSHOT -> com.foo:dependency_3:jar:1.2.3-SNAPSHOT -> com.foo.somegroup:dependency_4:jar:3.0.0
> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:180)
> at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:195)
> ... 16 more
> Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at com.foo:dependency_1:jar:1.2.3-SNAPSHOT -> com.foo:dependency_2:jar:1.2.3-SNAPSHOT -> com.foo:dependency_3:jar:1.2.3-SNAPSHOT -> com.foo.somegroup:dependency_4:jar:3.0.0
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:291)
> at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:316)
> at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:172)
> ... 17 more
> Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for com.foo.somegroup:dependency_4:jar:3.0.0
> at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:302)
> at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor(DefaultArtifactDescriptorReader.java:217)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.resolveCachedArtifactDescriptor(DefaultDependencyCollector.java:525)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.getArtifactDescriptorResult(DefaultDependencyCollector.java:509)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:409)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:363)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:351)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:494)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:363)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:351)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:494)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:363)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:351)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.doRecurse(DefaultDependencyCollector.java:494)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:458)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.processDependency(DefaultDependencyCollector.java:363)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.process(DefaultDependencyCollector.java:351)
> at org.eclipse.aether.internal.impl.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:254)
> ... 19 more
> Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact com.foo.somegroup:dependency_4:pom:3.0.0 from/to release-repository (https://nexus.foo.com/content/repositories/releases): /home/userid/.m2/repository/com/foo/somegroup/dependency_4/3.0.0/dependency_4-3.0.0.pom.part (No such file or directory)
> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:444)
> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts(DefaultArtifactResolver.java:246)
> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact(DefaultArtifactResolver.java:223)
> at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom(DefaultArtifactDescriptorReader.java:287)
> ... 38 more
> Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact com.foo.somegroup:dependency_4:pom:3.0.0 from/to release-repository (https://nexus.foo.com/content/repositories/releases): /home/userid/.m2/repository/com/foo/somegroup/dependency_4/3.0.0/dependency_4-3.0.0.pom.part (No such file or directory)
> at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed(ArtifactTransportListener.java:43)
> at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:355)
> at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run(RunnableErrorForwarder.java:67)
> at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute(BasicRepositoryConnector.java:581)
> at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get(BasicRepositoryConnector.java:249)
> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads(DefaultArtifactResolver.java:520)
> at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve(DefaultArtifactResolver.java:421)
> ... 41 more
> Caused by: java.io.FileNotFoundException: /home/userid/.m2/repository/com/foo/somegroup/dependency_4/3.0.0/dependency_4-3.0.0.pom.part (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:148)
> at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy(DefaultFileProcessor.java:136)
> at org.eclipse.aether.internal.impl.DefaultFileProcessor.move(DefaultFileProcessor.java:211)
> at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask(BasicRepositoryConnector.java:472)
> at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run(BasicRepositoryConnector.java:350)
> ... 46 more
> {noformat}
> * Note the '{{.part}}' extension to the missing pomfile. I see the same things with jars, wars, etc. '{{.part}}' is _always_ tacked on to the end of the artifact filename.
> * The dependency that Maven can't download -- and even the module that fails -- changes from one run to the next.
> * Subsequent builds attempts eventually download all dependencies to the local Maven cache, after which there is no problem building.
> Since I have to delete a substantial part of the local Maven cache for each build, I can't really count on the Maven cache.
> Using a single-thread build works fine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)