[iceberg] branch master updated: Core: Remove Usage of Sets.Union in Loop (#5114)

[ru...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

russellspitzer pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iceberg.git


The following commit(s) were added to refs/heads/master by this push:
     new 88a4d9e85 Core: Remove Usage of Sets.Union in Loop (#5114)
88a4d9e85 is described below

commit 88a4d9e85bf30c57d8ef0d2fc10315ba68c59360
Author: Russell Spitzer <rs...@apple.com>
AuthorDate: Wed Jun 22 16:44:06 2022 -0500

    Core: Remove Usage of Sets.Union in Loop (#5114)
    
    Previously the repeated calls to union could create a very large stack depth since each call ends up creating a set view. This can lead to StackOverflow issues with very large compaction jobs.
---
 .../org/apache/iceberg/actions/RewriteDataFilesCommitManager.java     | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesCommitManager.java b/core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesCommitManager.java
index 9a5cc4c94..277b48000 100644
--- a/core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesCommitManager.java
+++ b/core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesCommitManager.java
@@ -74,8 +74,8 @@ public class RewriteDataFilesCommitManager {
     Set<DataFile> rewrittenDataFiles = Sets.newHashSet();
     Set<DataFile> addedDataFiles = Sets.newHashSet();
     for (RewriteFileGroup group : fileGroups) {
-      rewrittenDataFiles = Sets.union(rewrittenDataFiles, group.rewrittenFiles());
-      addedDataFiles = Sets.union(addedDataFiles, group.addedFiles());
+      rewrittenDataFiles.addAll(group.rewrittenFiles());
+      addedDataFiles.addAll(group.addedFiles());
     }
 
     RewriteFiles rewrite = table.newRewrite().validateFromSnapshot(startingSnapshotId);