You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by GitBox <gi...@apache.org> on 2021/12/28 16:32:09 UTC
[GitHub] [maven-resolver] ibabiankou opened a new pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
ibabiankou opened a new pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142
Updates `DefaultDependencyCollector` to process descriptors concurrently.
Updated implementation does breadth-first traversal of the dependency graph and processes each dependency in a separate thread.
Order of the children nodes then restored to match order in descriptor of parent.
By default uses 5 threads, which can be changed using either `maven.descriptor.threads` or `maven.artifact.threads` property.
All the unit and integration tests are green. The change was also tested locally on a reasonable codebase (~1100 modules with lots dependencies) with 24 modules processed concurrently (-T1C).
**Important note**: a few times out of dosens of tests I observed an exception that seems to be related to the change (stack trace attached), however I couldn't really piece it together how it's possible, so ideas are welcome.
<details><summary>Stack Trace</summary>
<p>
```java
[ERROR] Failed to execute goal on project project-name: Could not resolve dependencies for project group-id:project-name:jar:2.16.1-SNAPSHOT: Failed to collect dependencies at dep-group-id:dep-art-id:jar:v2.9.0: Failed to read artifact descriptor for dep-group-id:dep-art-id:jar:v2.9.0: Could not transfer artifact dep-group-id:dep-art-id:pom:v2.9.0 from/to repo-name (https://private.nexus.url.com/repository/maven-public): /path/to/local_repo/v2.9.0/dep-art-id-v2.9.0.pom.part (No such file or directory) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project project-name: Could not resolve dependencies for project group-id:project-name:jar:2.16.1-SNAPSHOT: Failed to collect dependencies at dep-group-id:dep-art-id:jar:v2.9.0
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:269)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:833)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project group-id:project-name:jar:2.16.1-SNAPSHOT: Failed to collect dependencies at dep-group-id:dep-art-id:jar:v2.9.0
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:179)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:833)
Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at dep-group-id:dep-art-id:jar:v2.9.0
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies (DefaultDependencyCollector.java:299)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies (DefaultRepositorySystem.java:284)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:170)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:833)
Caused by: org.eclipse.aether.resolution.ArtifactDescriptorException: Failed to read artifact descriptor for dep-group-id:dep-art-id:jar:v2.9.0
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:259)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:175)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.resolveCachedArtifactDescriptor (DefaultDependencyCollector.java:617)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.getArtifactDescriptorResult (DefaultDependencyCollector.java:602)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency (DefaultDependencyCollector.java:491)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.access$000 (DefaultDependencyCollector.java:87)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector$1.run (DefaultDependencyCollector.java:395)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:833)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact dep-group-id:dep-art-id:pom:v2.9.0 from/to repo-name (https://private.nexus.url.com/repository/maven-public): /path/to/local_repo/v2.9.0/dep-art-id-v2.9.0.pom.part (No such file or directory)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:425)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:229)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact (DefaultArtifactResolver.java:207)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:244)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:175)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.resolveCachedArtifactDescriptor (DefaultDependencyCollector.java:617)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.getArtifactDescriptorResult (DefaultDependencyCollector.java:602)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency (DefaultDependencyCollector.java:491)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.access$000 (DefaultDependencyCollector.java:87)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector$1.run (DefaultDependencyCollector.java:395)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:833)
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact dep-group-id:dep-art-id:pom:v2.9.0 from/to repo-name (https://private.nexus.url.com/repository/maven-public): /path/to/local_repo/v2.9.0/dep-art-id-v2.9.0.pom.part (No such file or directory)
at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed (ArtifactTransportListener.java:52)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:369)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute (BasicRepositoryConnector.java:628)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get (BasicRepositoryConnector.java:262)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads (DefaultArtifactResolver.java:514)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:402)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:229)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact (DefaultArtifactResolver.java:207)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:244)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:175)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.resolveCachedArtifactDescriptor (DefaultDependencyCollector.java:617)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.getArtifactDescriptorResult (DefaultDependencyCollector.java:602)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency (DefaultDependencyCollector.java:491)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.access$000 (DefaultDependencyCollector.java:87)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector$1.run (DefaultDependencyCollector.java:395)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:833)
Caused by: java.io.FileNotFoundException: /path/to/local_repo/v2.9.0/dep-art-id-v2.9.0.pom.part (No such file or directory)
at java.io.FileInputStream.open0 (Native Method)
at java.io.FileInputStream.open (FileInputStream.java:216)
at java.io.FileInputStream.<init> (FileInputStream.java:157)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy (DefaultFileProcessor.java:163)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy (DefaultFileProcessor.java:151)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.move (DefaultFileProcessor.java:252)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask (BasicRepositoryConnector.java:482)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:364)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$DirectExecutor.execute (BasicRepositoryConnector.java:628)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector.get (BasicRepositoryConnector.java:262)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.performDownloads (DefaultArtifactResolver.java:514)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:402)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:229)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifact (DefaultArtifactResolver.java:207)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.loadPom (DefaultArtifactDescriptorReader.java:244)
at org.apache.maven.repository.internal.DefaultArtifactDescriptorReader.readArtifactDescriptor (DefaultArtifactDescriptorReader.java:175)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.resolveCachedArtifactDescriptor (DefaultDependencyCollector.java:617)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.getArtifactDescriptorResult (DefaultDependencyCollector.java:602)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.processDependency (DefaultDependencyCollector.java:491)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.access$000 (DefaultDependencyCollector.java:87)
at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector$1.run (DefaultDependencyCollector.java:395)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:539)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:635)
at java.lang.Thread.run (Thread.java:833)
```
</p>
</details>
- Reviewing individual commits might be easier.
- Some unit tests will be added if there are no objections WRT overall approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] michael-o commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
michael-o commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1002230454
I am very certain that the stack trace is a result of a race condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] michael-o commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
michael-o commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1023182162
>
Issue isn't related. This is a vanilla concurrency issue which will go away with Maven 3.9.0 by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] michael-o commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
michael-o commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1002264907
@ibabiankou I need to think about this because from the top of my head I don't understand why we need advisory file lock at all. We have now decent `SyncContext` implementations which should solve the problem for us.
@cstamas Please share your opinion when you are back to normal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] michael-o commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
michael-o commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1002266511
Does this intersect with https://issues.apache.org/jira/browse/MRESOLVER-133?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] ibabiankou commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
ibabiankou commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1002591300
π€― There are several improvements related to the same functionality that seems to be active at the moment:
1. https://issues.apache.org/jira/browse/MRESOLVER-7 (this PR)
2. https://issues.apache.org/jira/browse/MRESOLVER-133 (No PR)
3. https://issues.apache.org/jira/browse/MRESOLVER-228 (https://github.com/apache/maven-resolver/pull/136)
Lets have a look if we can reconcile them :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] michael-o commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
michael-o commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1002619756
@ibabiankou Thank you for looking into this. I see three mostly independent issues to be solved:
* Move from DFS to BFS which is a logical change and maybe may lead a difference dependency tree
* Skip already visitited artifacts which is a logical change; expected not to change the dependency tree
* Download POM artifacts in parallel which is a purely technical improvement and does not affect dependency tree resolution at all
Does this make sense?
Therefore, I'd like to discuss the first two and then the parallel approach. I don't want to review all of them in parallel (no pun intended)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] ibabiankou commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
ibabiankou commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1002661398
@michael-o They are quite independent indeed. They all related to the same area in the code, so even if considered and implemented separately I doubt it will be possible to revert them individually.
>* Move from DFS to BFS which is a logical change and maybe may lead a difference dependency tree
Are you talking here about the entire https://issues.apache.org/jira/browse/MRESOLVER-133 (DFS > BFS and changes to the version selection logic) or DFS to BFS change only? I believe it would be easier to consider them separately as version selection order can be changed separately.
I can see how version selection change might result to a different dependency tree, however DFS > BFS alone should not have any impact, right? π€
>* Skip already visitited artifacts which is a logical change; expected not to change the dependency tree
>* Download POM artifacts in parallel which is a purely technical improvement and does not affect dependency tree resolution at all
Agree.
I'll try to help with discussions and review, because I've already spent some time on this and would love to see these three changes to be implemented and released sooner rather than later π
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] cstamas commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
cstamas commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1026093783
pls incorporate latest master changes (whether rebase+squash+force push or just merge)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] Tibor17 commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
Tibor17 commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1023218695
Don't use the old Java Utils model.
`Collections.synchronizedList( new ArrayList<DependencyNode>( 0 ) )`
This is still thread-unsafe due to the `ArrayList` is unsafe.
Every implementatino of `ArrayList` has two variables (int size, Object[]). Due to this implementation is not safe and due to the writer thread created the object, there is no memorry barrier which would force the thread to write the CPU cache to the main memory and thus there is no guarantee that a reader thread would safely read these variales as initialized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] ibabiankou commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
ibabiankou commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1022263329
Not that it's solid proof, but got a very similar exception today while using the stock maven 3.8.4 π€
<details><summary>Stack Trace</summary>
<p>
```java
[ERROR] Failed to execute goal on project dummy-artifact: Could not resolve dependencies for project dummy-group:dummy-artifact:jar:63.0-SNAPSHOT: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project dummy-artifact: Could not resolve dependencies for project dummy-group:dummy-artifact:jar:63.0-SNAPSHOT: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:269)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: org.apache.maven.project.DependencyResolutionException: Could not resolve dependencies for project dummy-group:dummy-artifact:jar:63.0-SNAPSHOT: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:198)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: org.eclipse.aether.resolution.DependencyResolutionException: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:357)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:191)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:425)
at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:229)
at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:340)
at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:191)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243)
at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147)
at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:196)
at org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call (MultiThreadedBuilder.java:186)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:515)
at java.util.concurrent.FutureTask.run (FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: org.eclipse.aether.transfer.ArtifactTransferException: Could not transfer artifact io.sentry:sentry:jar:5.5.3 from/to nexus-server (https://nexus.at-our-company.com/repository/maven-public): /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed (ArtifactTransportListener.java:52)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:369)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
Caused by: java.io.FileNotFoundException: /Users/user/.m2/repository/io/sentry/sentry/5.5.3/sentry-5.5.3.jar.part (No such file or directory)
at java.io.FileInputStream.open0 (Native Method)
at java.io.FileInputStream.open (FileInputStream.java:219)
at java.io.FileInputStream.<init> (FileInputStream.java:157)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy (DefaultFileProcessor.java:163)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.copy (DefaultFileProcessor.java:151)
at org.eclipse.aether.internal.impl.DefaultFileProcessor.move (DefaultFileProcessor.java:252)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$GetTaskRunner.runTask (BasicRepositoryConnector.java:482)
at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:364)
at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:628)
at java.lang.Thread.run (Thread.java:829)
```
</p>
</details>
As discussed earlier, I will revive this PR once https://github.com/apache/maven-resolver/pull/144 and updated version of https://github.com/apache/maven-resolver/pull/136 are merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] ibabiankou commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
ibabiankou commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1002586928
> Does this intersect with https://issues.apache.org/jira/browse/MRESOLVER-133?
@michael-o Only about the switching to BFS strategy, however itβs not the biggest part of the change. In fact the DFS > BFS change is irrelevant for that ticket as far as I can tell. The goal there is to change which version out of the version range is selected.
**Now**
1. Descriptors of all versions in all ranges resolved and added to the dependency graph.
2. Dependency graph is transformed to resolve version conflicts (among other things).
**Implied**
1. Use the first newest resolvable descriptor; _(it makes sense to me, in fact this is almost [how the version of the root dependency is resolved](https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java#L187) except it does not proceed to older versions if newest can't be resolved)._
2. Dependency graph is transformed to resolve version conflicts (among other things); _(still relevant since in other parts of the graph different version [range] might be used)_
I'd say the ticket title and description is a bit misleading, but the improvement itself does make sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] ibabiankou closed pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
ibabiankou closed pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] ibabiankou commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
ibabiankou commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1027133750
@Tibor17 Thanks for pointing this out, however, I'm a bit puzzled as I don't see a good alternative here π€ Do you have any ideas from the top of your head?
@cstamas The plan is to first focus on reviewing and merging #144, then work on either this one or #136. I'll close this one for now and re-open or open another one later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] michael-o commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
michael-o commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1002665297
> @michael-o They are quite independent indeed. They all related to the same area in the code, so even if considered and implemented separately I doubt it will be possible to revert them individually.
Yes, they are related, but we can work on them *serially*.
> > * Move from DFS to BFS which is a logical change and maybe may lead a difference dependency tree
>
> Are you talking here about the entire https://issues.apache.org/jira/browse/MRESOLVER-133 (DFS > BFS and changes to the version selection logic) or DFS to BFS change only? I believe it would be easier to consider them separately as version selection order can be changed separately. I can see how version selection change might result to a different dependency tree, however DFS > BFS alone should not have any impact, right? π€
Clearly, I want them separately. pure DFS to BFS change only w/o any behavior change. Then discuss selection order which intersects with other issues as well.
> > * Skip already visitited artifacts which is a logical change; expected not to change the dependency tree
> > * Download POM artifacts in parallel which is a purely technical improvement and does not affect dependency tree resolution at all
>
> Agree.
>
> I'll try to help with discussions and review, because I've already spent some time on this and would love to see these three changes to be implemented and released sooner rather than later π
Yes, I already thought so. I am happy to review and merge upstream as long as:
* My time spare allows
* Another dev (e.g., @cstamas) reviews
* A PR is focused on one change only
* Can be completely understood with all of its implications
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [maven-resolver] ibabiankou commented on pull request #142: [MRESOLVER-7] Download dependency POMs in parallel
Posted by GitBox <gi...@apache.org>.
ibabiankou commented on pull request #142:
URL: https://github.com/apache/maven-resolver/pull/142#issuecomment-1002256947
@michael-o I'm absolutely certain of that too. I'm not sure however what is the best way to solve it, especially because I don't have a reliable way to reproduce it :disappointed:
Looking at the implementation of the `BasicRepositoryConnector$GetTaskRunner` each artifact download should create a [PartialFile](https://github.com/apache/maven-resolver/blob/master/maven-resolver-connector-basic/src/main/java/org/eclipse/aether/connector/basic/BasicRepositoryConnector.java#L442) which also creates a corresponding [LockFile](https://github.com/apache/maven-resolver/blob/master/maven-resolver-connector-basic/src/main/java/org/eclipse/aether/connector/basic/PartialFile.java#L268) which actually [blocks until it gets an exclusive lock](https://github.com/apache/maven-resolver/blob/master/maven-resolver-connector-basic/src/main/java/org/eclipse/aether/connector/basic/PartialFile.java#L114) :detective: It makes me wonder whether there is an issue with that logic :thinking:
To fix it I can think of modifying [getArtifactDescriptorResult](https://github.com/ibabiankou/maven-resolver/blob/MRESOLVER-7/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java#L600) to ensure it allows single call for the same artifact at a time ([something like this](https://github.com/ibabiankou/maven-resolver/commit/4a906a1ed8c0e0358275981aa243fdfe5dab1d01)). But that feels a bit wrong place to solve this.
Ideally I would put that kind of logic in the `DefaultArtifactResolver`, but that would be too big of a change to make in the scope of this PR :sweat_smile:
What do you think @michael-o? Does [this approach](https://github.com/ibabiankou/maven-resolver/commit/4a906a1ed8c0e0358275981aa243fdfe5dab1d01) make sense to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org