You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "gnodet (via GitHub)" <gi...@apache.org> on 2026/06/28 07:00:12 UTC

[PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

gnodet opened a new pull request, #1938:
URL: https://github.com/apache/maven-resolver/pull/1938

   ## Summary
   
   - **Auto-selection heuristic** (new default `"auto"` mode): estimates Path tree memory cost from the conflict-ID map size (`~200 bytes × nodeCount`) and falls back to `ClassicConflictResolver` if the parallel tree would exceed 25% of available heap. This prevents OOM on very large dependency graphs (e.g. ~1000-module projects with millions of transitive nodes) while preserving the faster O(N) path resolver for projects that fit comfortably in memory.
   
   - **Lazy right-sized children lists**: `Path.children` starts as `null` (not an empty `ArrayList`) and is only initialized in `addChildren()` with exactly the right capacity. Leaf nodes (~60-70% of all nodes) never allocate a list at all, saving ~40 bytes per leaf (the `ArrayList` object header + default `Object[10]` backing array).
   
   - **`outOfScope` flag replaces `LinkedHashSet` partition entries**: `moveOutOfScope()` now sets a `boolean` flag instead of removing from a `LinkedHashSet`, eliminating the `HashMap.Node` overhead (~48 bytes per `Path` entry). Active paths are filtered at query time with a simple boolean check.
   
   ### Memory savings estimate
   
   For a graph with N nodes where ~65% are leaves:
   | Optimization | Per-node saving | Total for 5M nodes |
   |---|---|---|
   | Lazy children (leaves) | ~40 bytes | ~130 MB |
   | `outOfScope` flag | ~48 bytes | ~240 MB |
   | Right-sized ArrayList | ~40 bytes (non-leaves) | ~70 MB |
   | **Total** | | **~440 MB** |
   
   Additionally, the auto-selection heuristic avoids the full Path tree (~1 GB for 5M nodes) entirely when memory is tight, falling back to the in-place classic resolver.
   
   ## Test plan
   
   - [x] All 438 existing `maven-resolver-util` tests pass (parameterized over both Path and Classic resolvers)
   - [ ] CI build passes
   - [ ] Manual verification on a large multi-module project
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "seregamorph (via GitHub)" <gi...@apache.org>.
seregamorph commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4825881088

   Few comments:
   * indeed, the number of nested recursive calls inside of `gatherCRNodes` is high, but there is no StackOverflowError (as it happens with way deeper stack), it just means that if the complexity is high in each nested iteration, this can lead to OOM/high CPU while traversing too many tree elements
   * The `build hang` - almost agree there is not caused by this particular PR, but still deserves attention 🙏 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "gnodet (via GitHub)" <gi...@apache.org>.
gnodet commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4829329160

   _Claude Code on behalf of Guillaume Nodet_
   
   Ran an independent benchmark with the reproducer project (814-module `server_subset`) to validate the changes.
   
   ### Results
   
   | Configuration | Heap | Time |
   |---|---|---|
   | Maven 3.9.16 (baseline) | 768m | 135,724ms (~2m16s) |
   | 3.10.0-SNAPSHOT (patched, auto) | 768m | 135,082ms (~2m15s) |
   | Maven 3.9.16 (baseline) | 16g | 135,724ms (~2m16s) |
   | 3.10.0-SNAPSHOT (patched, auto) | 16g | 135,082ms (~2m15s) |
   | 3.10.0-SNAPSHOT (patched, `-Daether.conflictResolver.impl=path`) | 16g | 91,435ms (~1m31s) |
   
   ### Observations
   
   - With `auto` mode (the default), both 3.9.16 and 3.10.0-SNAPSHOT perform identically (~2m15s) regardless of heap size — the auto heuristic falls back to `ClassicConflictResolver`.
   - When forcing `path` resolver with 16g heap, resolution completes in ~1m31s — **33% faster** than the classic resolver.
   - The `auto` heuristic does not select `path` even with 16g because the 25% threshold (`available / (4 * 200)` ≈ 21M nodes) is exceeded by the total tree node count. The PR mentions a single module with 7,522 unique nodes producing 1,190,821 tree nodes (158× expansion factor) — across 814 modules, the total easily exceeds the threshold.
   - This suggests the 25% threshold may be too conservative for projects where the `path` resolver would provide significant speedup. It might be worth considering either raising the threshold or making it configurable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "seregamorph (via GitHub)" <gi...@apache.org>.
seregamorph commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4826451018

   Checked 0b4ca58b6e4309e25eee334296a29ab90b0039f0
   No, the problem is not solved. Please try the reproducing project - it also still shows the performance issue (compare the build speed with 3.9)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "seregamorph (via GitHub)" <gi...@apache.org>.
seregamorph commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4825719135

   The build gets stuck. Some stack traces:
   ```
   "mvn-turbo-builder-runner-board" #56 prio=5 os_prio=31 cpu=54709.59ms elapsed=618.98s tid=0x0000000853d73600 nid=0xd603 waiting on condition  [0x0000000178b41000]
      java.lang.Thread.State: WAITING (parking)
   	at jdk.internal.misc.Unsafe.park(java.base@17.0.19/Native Method)
   	- parking to wait for  <0x0000000b4cfd4290> (a java.util.concurrent.CompletableFuture$Signaller)
   	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.19/LockSupport.java:211)
   	at java.util.concurrent.CompletableFuture$Signaller.block(java.base@17.0.19/CompletableFuture.java:1864)
   	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.19/ForkJoinPool.java:3465)
   	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.19/ForkJoinPool.java:3436)
   	at java.util.concurrent.CompletableFuture.waitingGet(java.base@17.0.19/CompletableFuture.java:1898)
   	at java.util.concurrent.CompletableFuture.get(java.base@17.0.19/CompletableFuture.java:2072)
   	at org.eclipse.aether.internal.impl.collect.bf.BfDependencyCollector.processDependency(BfDependencyCollector.java:257)
   	at org.eclipse.aether.internal.impl.collect.bf.BfDependencyCollector.doCollectDependencies(BfDependencyCollector.java:222)
   	at org.eclipse.aether.internal.impl.collect.DependencyCollectorDelegate.collectDependencies(DependencyCollectorDelegate.java:260)
   	at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:79)
   	at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:247)
   	at org.apache.maven.enforcer.rules.dependency.ResolverUtil.resolveTransitiveDependencies(ResolverUtil.java:149)
   	at org.apache.maven.enforcer.rules.dependency.ResolverUtil.resolveTransitiveDependenciesVerbose(ResolverUtil.java:85)
   	at org.apache.maven.enforcer.rules.dependency.DependencyConvergence.execute(DependencyConvergence.java:68)
   	at org.apache.maven.plugins.enforcer.EnforceMojo.executeRuleNew(EnforceMojo.java:351)
   	at org.apache.maven.plugins.enforcer.EnforceMojo.executeRule(EnforceMojo.java:325)
   	at org.apache.maven.plugins.enforcer.EnforceMojo.execute(EnforceMojo.java:248)
   	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:127)
   	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2(MojoExecutor.java:336)
   	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute(MojoExecutor.java:324)
   	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:220)
   	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:181)
   	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000(MojoExecutor.java:79)
   	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run(MojoExecutor.java:169)
   	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute(DefaultMojosExecutionStrategy.java:39)
   	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:166)
   	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:106)
   ```
   
   While all bfcollector threads are sleeping:
   ```
   "BfDependencyCollector-1-1" #46 daemon prio=5 os_prio=31 cpu=630.15ms elapsed=619.24s tid=0x0000000852760600 nid=0xc903 waiting on condition  [0x00000001776ca000]
      java.lang.Thread.State: WAITING (parking)
   	at jdk.internal.misc.Unsafe.park(java.base@17.0.19/Native Method)
   	- parking to wait for  <0x0000000b5dacfec8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.park(java.base@17.0.19/LockSupport.java:341)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(java.base@17.0.19/AbstractQueuedSynchronizer.java:506)
   	at java.util.concurrent.ForkJoinPool.unmanagedBlock(java.base@17.0.19/ForkJoinPool.java:3465)
   	at java.util.concurrent.ForkJoinPool.managedBlock(java.base@17.0.19/ForkJoinPool.java:3436)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@17.0.19/AbstractQueuedSynchronizer.java:1630)
   	at java.util.concurrent.LinkedBlockingQueue.take(java.base@17.0.19/LinkedBlockingQueue.java:435)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@17.0.19/ThreadPoolExecutor.java:1062)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.19/ThreadPoolExecutor.java:1122)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.19/ThreadPoolExecutor.java:635)
   	at java.lang.Thread.run(java.base@17.0.19/Thread.java:840)
   ```
   
   Worth to mention:
   * there is no OOM
   * the options are
   ```
   # AFAIK default now
   -Daether.dependencyCollector.impl=bf
   -Daether.dependencyCollector.bf.threads=10
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "seregamorph (via GitHub)" <gi...@apache.org>.
seregamorph commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4825792678

   After rerunning the build I have OOM problems. The implementation consumes memory quite actively (for 3.9.16 it's enough to have only 9g):
   
   <img width="560" height="406" alt="Screenshot 2026-06-28 at 12 20 26" src="https://github.com/user-attachments/assets/512da0a2-0260-4747-a909-4f58c44d2f28" />
   
   At some points the process gets stuck, the stack trace is:
   ```
   "mvn-turbo-builder-data-migration-between-regions-temporal-organizations-integration-test" #45 prio=5 os_prio=31 cpu=66339.06ms elapsed=229.64s tid=0x000000099b6cd800 nid=0xda03 runnable  [0x00000001734f5000]
      java.lang.Thread.State: RUNNABLE
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$Path.addChildren(PathConflictResolver.java:680)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$Path.access$2200(PathConflictResolver.java:386)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:350)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.gatherCRNodes(PathConflictResolver.java:352)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.build(PathConflictResolver.java:339)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.<init>(PathConflictResolver.java:328)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver$State.<init>(PathConflictResolver.java:253)
   	at org.eclipse.aether.util.graph.transformer.PathConflictResolver.transformGraph(PathConflictResolver.java:168)
   	at org.eclipse.aether.util.graph.transformer.ConflictResolver.transformGraph(ConflictResolver.java:290)
   	at org.eclipse.aether.util.graph.transformer.ChainedDependencyGraphTransformer.transformGraph(ChainedDependencyGraphTransformer.java:72)
   	at org.eclipse.aether.internal.impl.collect.DependencyCollectorDelegate.collectDependencies(DependencyCollectorDelegate.java:284)
   	at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies(DefaultDependencyCollector.java:79)
   	at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies(DefaultRepositorySystem.java:247)
   	at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve(DefaultProjectDependenciesResolver.java:154)
   	at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies(LifecycleDependencyResolver.java:231)
   	at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies(LifecycleDependencyResolver.java:140)
   ```
   
   Regarding the stuck state - I've already seen this, it happens once in a while. But this actually means there is a wrong multi-thread implementation somewhere and at scale these problems will happen quite often.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "seregamorph (via GitHub)" <gi...@apache.org>.
seregamorph commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4834523600

   LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "gnodet (via GitHub)" <gi...@apache.org>.
gnodet commented on code in PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#discussion_r3487879095


##########
maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/transformer/ConflictResolver.java:
##########
@@ -258,22 +264,58 @@ public ConflictResolver(
     }
 
     @Override
+    @SuppressWarnings("unchecked")
     public DependencyNode transformGraph(DependencyNode node, DependencyGraphTransformationContext context)
             throws RepositoryException {
         String cf = ConfigUtils.getString(
                 context.getSession(), DEFAULT_CONFLICT_RESOLVER_IMPL, CONFIG_PROP_CONFLICT_RESOLVER_IMPL);
         ConflictResolver delegate;
-        if (PATH_CONFLICT_RESOLVER.equals(cf)) {
+        if (AUTO_CONFLICT_RESOLVER.equals(cf)) {
+            delegate = selectConflictResolver(node, context);
+        } else if (PATH_CONFLICT_RESOLVER.equals(cf)) {
             delegate = new PathConflictResolver(versionSelector, scopeSelector, optionalitySelector, scopeDeriver);
         } else if (CLASSIC_CONFLICT_RESOLVER.equals(cf)) {
             delegate = new ClassicConflictResolver(versionSelector, scopeSelector, optionalitySelector, scopeDeriver);
         } else {
             throw new IllegalArgumentException("Unknown conflict resolver: " + cf + "; known are "
-                    + Arrays.asList(PATH_CONFLICT_RESOLVER, CLASSIC_CONFLICT_RESOLVER));
+                    + Arrays.asList(AUTO_CONFLICT_RESOLVER, PATH_CONFLICT_RESOLVER, CLASSIC_CONFLICT_RESOLVER));
         }
         return delegate.transformGraph(node, context);
     }
 
+    /**
+     * Selects the most appropriate conflict resolver based on graph size and available memory.
+     * <p>
+     * PathConflictResolver builds a parallel tree of Path objects that costs ~200 bytes per node.
+     * For very large dependency graphs (millions of nodes), this can exhaust the heap.

Review Comment:
   Fixed in 0b4ca58. Added 5 unit tests covering the delegating `ConflictResolver` dispatch: explicit `"auto"` config, explicit `"path"` config, explicit `"classic"` config, unknown config rejection (`IllegalArgumentException`), and default config (no property set, exercises the auto-selection code path). All 443 tests pass.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "cstamas (via GitHub)" <gi...@apache.org>.
cstamas commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4834849251

   Maybe add this one too https://gist.github.com/cstamas/99f760aa13d738f02712accfd4a49c29


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "gnodet (via GitHub)" <gi...@apache.org>.
gnodet commented on code in PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#discussion_r3487878862


##########
maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/transformer/PathConflictResolver.java:
##########
@@ -170,28 +167,33 @@ public DependencyNode transformGraph(DependencyNode node, DependencyGraphTransfo
                 scopeDeriver.getInstance(node, context),
                 optionalitySelector.getInstance(node, context),
                 conflictIds,
+                sortedConflictIds.size(),
                 node);
 
         // loop over topographically sorted conflictIds
         for (String conflictId : sortedConflictIds) {
-            // paths in given conflict group to consider
-            Set<Path> paths = state.partitions.get(conflictId);
-            if (paths.isEmpty()) {
+            // paths in given conflict group to consider; filter out those moved out of scope
+            List<Path> allPaths = state.partitions.get(conflictId);
+            List<Path> activePaths = new ArrayList<>(allPaths.size());
+            List<ConflictItem> items = new ArrayList<>(allPaths.size());
+            for (Path p : allPaths) {
+                if (!p.outOfScope) {
+                    activePaths.add(p);
+                    items.add(new ConflictItem(p));
+                }
+            }
+            if (activePaths.isEmpty()) {

Review Comment:
   Fixed in 0b4ca58. Partition entries are now replaced with the filtered `activePaths` list after each conflict group, releasing references to out-of-scope paths and their detached subtrees for GC during resolution. Additionally, `moveOutOfScope()` now nulls children references on marked nodes to break reference chains, and `conflictItemCount` is tracked inline during the main loop instead of re-scanning partitions post-resolution.



##########
maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/transformer/PathConflictResolver.java:
##########
@@ -610,43 +643,53 @@ private boolean isDirectDependencyOnPathToRoot(Artifact artifact) {
          * verbosity mode we remove "redundant" nodes (of a range) leaving only "winner equal" loser, that have same GACEV as winner.
          */
         private int relatedSiblingsCount(Artifact artifact, Path parent) {
-            String ga = artifact.getGroupId() + ":" + artifact.getArtifactId();
-            return Math.toIntExact(parent.children.stream()
-                    .map(n -> n.dn.getArtifact())
-                    .filter(a -> ga.equals(a.getGroupId() + ":" + a.getArtifactId()))
-                    .count());
+            if (parent.children == null) {
+                return 0;
+            }
+            String groupId = artifact.getGroupId();
+            String artifactId = artifact.getArtifactId();
+            int count = 0;
+            for (Path n : parent.children) {
+                Artifact a = n.dn.getArtifact();
+                if (Objects.equals(groupId, a.getGroupId()) && Objects.equals(artifactId, a.getArtifactId())) {
+                    count++;
+                }
+            }
+            return count;
         }
 
         /**
-         * Removes this and all child {@link Path} nodes from winner selection scope; essentially marks whole subtree
+         * Marks this and all child {@link Path} nodes as out of scope; essentially marks whole subtree
          * from "this and below" as loser, to not be considered in subsequent winner selections.
-         * Uses {@link LinkedHashSet} for O(1) removal instead of the previous O(N) ArrayList.remove.
+         * Uses a boolean flag instead of removing from partition collections, avoiding the per-entry
+         * overhead of LinkedHashSet (~48 bytes/entry). Out-of-scope paths are filtered at query time.
          */
         private void moveOutOfScope() {
-            this.state.partitions.get(this.conflictId).remove(this);
-            for (Path child : this.children) {
-                child.moveOutOfScope();
+            this.outOfScope = true;
+            if (this.children != null) {
+                for (Path child : this.children) {
+                    child.moveOutOfScope();

Review Comment:
   Fixed in 0b4ca58. Converted `moveOutOfScope()` to use an explicit `ArrayList` stack instead of recursion, consistent with the iterative `gatherCRNodes()` approach. The iterative version also nulls children references on out-of-scope nodes to accelerate GC of detached subtrees.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "cstamas (via GitHub)" <gi...@apache.org>.
cstamas commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4829648157

   So, then basically gigantic projects are doomed to use classic (if they insist on same memory amount; no speedup)?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#discussion_r3487784159


##########
maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/transformer/PathConflictResolver.java:
##########
@@ -170,28 +167,33 @@ public DependencyNode transformGraph(DependencyNode node, DependencyGraphTransfo
                 scopeDeriver.getInstance(node, context),
                 optionalitySelector.getInstance(node, context),
                 conflictIds,
+                sortedConflictIds.size(),
                 node);
 
         // loop over topographically sorted conflictIds
         for (String conflictId : sortedConflictIds) {
-            // paths in given conflict group to consider
-            Set<Path> paths = state.partitions.get(conflictId);
-            if (paths.isEmpty()) {
+            // paths in given conflict group to consider; filter out those moved out of scope
+            List<Path> allPaths = state.partitions.get(conflictId);
+            List<Path> activePaths = new ArrayList<>(allPaths.size());
+            List<ConflictItem> items = new ArrayList<>(allPaths.size());
+            for (Path p : allPaths) {
+                if (!p.outOfScope) {
+                    activePaths.add(p);
+                    items.add(new ConflictItem(p));
+                }
+            }
+            if (activePaths.isEmpty()) {

Review Comment:
   `partitions` retains out-of-scope `Path` entries indefinitely. Even though `activePaths` filters them out, the backing `allPaths` list still strongly references losers (and entire detached subtrees in `Verbosity.NONE`/redundant `STANDARD`), which can prevent GC and raise peak memory — the opposite of the intended OOM mitigation. Consider compacting `allPaths` in-place while filtering (or otherwise dropping references to out-of-scope entries) so resolved/losing branches don’t stay retained via `partitions`.



##########
maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/transformer/ConflictResolver.java:
##########
@@ -258,22 +264,58 @@ public ConflictResolver(
     }
 
     @Override
+    @SuppressWarnings("unchecked")
     public DependencyNode transformGraph(DependencyNode node, DependencyGraphTransformationContext context)
             throws RepositoryException {
         String cf = ConfigUtils.getString(
                 context.getSession(), DEFAULT_CONFLICT_RESOLVER_IMPL, CONFIG_PROP_CONFLICT_RESOLVER_IMPL);
         ConflictResolver delegate;
-        if (PATH_CONFLICT_RESOLVER.equals(cf)) {
+        if (AUTO_CONFLICT_RESOLVER.equals(cf)) {
+            delegate = selectConflictResolver(node, context);
+        } else if (PATH_CONFLICT_RESOLVER.equals(cf)) {
             delegate = new PathConflictResolver(versionSelector, scopeSelector, optionalitySelector, scopeDeriver);
         } else if (CLASSIC_CONFLICT_RESOLVER.equals(cf)) {
             delegate = new ClassicConflictResolver(versionSelector, scopeSelector, optionalitySelector, scopeDeriver);
         } else {
             throw new IllegalArgumentException("Unknown conflict resolver: " + cf + "; known are "
-                    + Arrays.asList(PATH_CONFLICT_RESOLVER, CLASSIC_CONFLICT_RESOLVER));
+                    + Arrays.asList(AUTO_CONFLICT_RESOLVER, PATH_CONFLICT_RESOLVER, CLASSIC_CONFLICT_RESOLVER));
         }
         return delegate.transformGraph(node, context);
     }
 
+    /**
+     * Selects the most appropriate conflict resolver based on graph size and available memory.
+     * <p>
+     * PathConflictResolver builds a parallel tree of Path objects that costs ~200 bytes per node.
+     * For very large dependency graphs (millions of nodes), this can exhaust the heap.

Review Comment:
   The new default "auto" resolver selection path isn’t covered by unit tests (existing tests exercise `PathConflictResolver` and `ClassicConflictResolver` directly). Adding tests that validate both branches of the heuristic would help prevent regressions (e.g., by refactoring heap-availability and/or estimate calculation behind overridable helpers or injectable suppliers so tests can deterministically force either selection).



##########
maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/transformer/PathConflictResolver.java:
##########
@@ -610,43 +643,53 @@ private boolean isDirectDependencyOnPathToRoot(Artifact artifact) {
          * verbosity mode we remove "redundant" nodes (of a range) leaving only "winner equal" loser, that have same GACEV as winner.
          */
         private int relatedSiblingsCount(Artifact artifact, Path parent) {
-            String ga = artifact.getGroupId() + ":" + artifact.getArtifactId();
-            return Math.toIntExact(parent.children.stream()
-                    .map(n -> n.dn.getArtifact())
-                    .filter(a -> ga.equals(a.getGroupId() + ":" + a.getArtifactId()))
-                    .count());
+            if (parent.children == null) {
+                return 0;
+            }
+            String groupId = artifact.getGroupId();
+            String artifactId = artifact.getArtifactId();
+            int count = 0;
+            for (Path n : parent.children) {
+                Artifact a = n.dn.getArtifact();
+                if (Objects.equals(groupId, a.getGroupId()) && Objects.equals(artifactId, a.getArtifactId())) {
+                    count++;
+                }
+            }
+            return count;
         }
 
         /**
-         * Removes this and all child {@link Path} nodes from winner selection scope; essentially marks whole subtree
+         * Marks this and all child {@link Path} nodes as out of scope; essentially marks whole subtree
          * from "this and below" as loser, to not be considered in subsequent winner selections.
-         * Uses {@link LinkedHashSet} for O(1) removal instead of the previous O(N) ArrayList.remove.
+         * Uses a boolean flag instead of removing from partition collections, avoiding the per-entry
+         * overhead of LinkedHashSet (~48 bytes/entry). Out-of-scope paths are filtered at query time.
          */
         private void moveOutOfScope() {
-            this.state.partitions.get(this.conflictId).remove(this);
-            for (Path child : this.children) {
-                child.moveOutOfScope();
+            this.outOfScope = true;
+            if (this.children != null) {
+                for (Path child : this.children) {
+                    child.moveOutOfScope();

Review Comment:
   `moveOutOfScope()` is still recursive. Deep dependency chains can still trigger `StackOverflowError` when an early node becomes a loser (the subtree walk recurses once per level). Converting this to an explicit stack keeps the “no recursion” goal consistent with the new iterative `gatherCRNodes()` change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4836398117

   @cstamas Please assign appropriate label to PR according to the type of change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "cstamas (via GitHub)" <gi...@apache.org>.
cstamas merged PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "cstamas (via GitHub)" <gi...@apache.org>.
cstamas commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4826412162

   @seregamorph have you tried latest changes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "gnodet (via GitHub)" <gi...@apache.org>.
gnodet commented on PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#issuecomment-4829945948

   _Claude Code on behalf of Guillaume Nodet_
   
   Ran an independent benchmark with the reproducer project (814-module `server_subset`) to validate the changes. Each configuration was run 3 times to confirm consistency.
   
   ### Results
   
   **Maven 3.9.16 (baseline, classic resolver)** — stable across heap sizes:
   
   | Heap | Run 1 | Run 2 | Run 3 | Avg |
   |---|---|---|---|---|
   | 4g | 95.6s | 97.3s | 96.0s | **96.3s** |
   | 8g | 94.2s | 97.3s | 98.1s | **96.5s** |
   | 16g | 95.4s | 96.2s | 98.2s | **96.6s** |
   
   **3.10.0-SNAPSHOT (patched, auto mode selects path resolver)**:
   
   | Heap | Run 1 | Run 2 | Run 3 | Avg | Speedup |
   |---|---|---|---|---|---|
   | 4g | 74.9s | 73.0s | 73.2s | **73.7s** | **23%** |
   | 8g | 80.1s | 78.7s | 79.2s | **79.3s** | **18%** |
   | 16g | 83.7s | 83.4s | 87.8s | **85.0s** | **12%** |
   
   ### Observations
   
   - The patched path resolver is **12–23% faster** than the 3.9.16 classic resolver across all tested heap sizes.
   - The auto-selection heuristic correctly selects `path` for all 333 modules at every heap size tested (4g, 8g, 16g).
   - There is a minor slowdown with larger heaps for the path resolver (~11s between 4g and 16g), likely due to GC behavior with the parallel tree structure. The classic resolver is unaffected by heap size.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Reduce PathConflictResolver memory and auto-select resolver [maven-resolver]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1938:
URL: https://github.com/apache/maven-resolver/pull/1938#discussion_r3487910914


##########
maven-resolver-util/src/main/java/org/eclipse/aether/util/graph/transformer/ConflictResolver.java:
##########
@@ -258,22 +264,58 @@ public ConflictResolver(
     }
 
     @Override
+    @SuppressWarnings("unchecked")
     public DependencyNode transformGraph(DependencyNode node, DependencyGraphTransformationContext context)
             throws RepositoryException {
         String cf = ConfigUtils.getString(
                 context.getSession(), DEFAULT_CONFLICT_RESOLVER_IMPL, CONFIG_PROP_CONFLICT_RESOLVER_IMPL);
         ConflictResolver delegate;
-        if (PATH_CONFLICT_RESOLVER.equals(cf)) {
+        if (AUTO_CONFLICT_RESOLVER.equals(cf)) {
+            delegate = selectConflictResolver(node, context);
+        } else if (PATH_CONFLICT_RESOLVER.equals(cf)) {
             delegate = new PathConflictResolver(versionSelector, scopeSelector, optionalitySelector, scopeDeriver);
         } else if (CLASSIC_CONFLICT_RESOLVER.equals(cf)) {
             delegate = new ClassicConflictResolver(versionSelector, scopeSelector, optionalitySelector, scopeDeriver);
         } else {
             throw new IllegalArgumentException("Unknown conflict resolver: " + cf + "; known are "
-                    + Arrays.asList(PATH_CONFLICT_RESOLVER, CLASSIC_CONFLICT_RESOLVER));
+                    + Arrays.asList(AUTO_CONFLICT_RESOLVER, PATH_CONFLICT_RESOLVER, CLASSIC_CONFLICT_RESOLVER));
         }
         return delegate.transformGraph(node, context);
     }
 
+    /**
+     * Selects the most appropriate conflict resolver based on graph size and available memory.
+     * <p>
+     * PathConflictResolver builds a parallel tree of Path objects that costs ~200 bytes per node.
+     * For very large dependency graphs (millions of nodes), this can exhaust the heap.
+     * In such cases, ClassicConflictResolver is used instead — it works in-place with no parallel
+     * structure, trading O(N²) worst-case time for O(1) extra space.
+     */
+    @SuppressWarnings("unchecked")
+    private ConflictResolver selectConflictResolver(DependencyNode node, DependencyGraphTransformationContext context)
+            throws RepositoryException {
+        // Ensure conflict IDs are computed — both implementations need this anyway
+        if (context.get(TransformationContextKeys.SORTED_CONFLICT_IDS) == null) {
+            new ConflictIdSorter().transformGraph(node, context);
+        }
+
+        Map<DependencyNode, String> conflictIds =
+                (Map<DependencyNode, String>) context.get(TransformationContextKeys.CONFLICT_IDS);
+        int nodeCount = conflictIds != null ? conflictIds.size() : 0;
+
+        // Estimate Path tree memory: ~200 bytes per Path (object + right-sized children list + partition entry)
+        long pathTreeEstimate = (long) nodeCount * 200;

Review Comment:
   The auto-selection heuristic uses `conflictIds.size()` as `nodeCount`, but `conflictIds` is built from an `IdentityHashMap` of *unique* `DependencyNode` instances (see `ConflictMarker`), while `PathConflictResolver` allocates a `Path` per *occurrence/path* (a node instance can appear under multiple parents, e.g. shared subgraphs/cycles). This can make `pathTreeEstimate` a lower bound and may still select `PathConflictResolver` in cases where the actual Path tree is much larger, undermining the OOM-avoidance goal. Consider either using a more conservative safety factor or improving the estimate to account for node re-use/graph sharing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@maven.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org