You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amogh-jahagirdar (via GitHub)" <gi...@apache.org> on 2025/12/23 17:23:47 UTC

[PR] Core: Small cleanup in MergingSnapshotProducer cleanUncommittedAppends [iceberg]

amogh-jahagirdar opened a new pull request, #14923:
URL: https://github.com/apache/iceberg/pull/14923

   Was going through MergingSnapshotProducer code and noticed that there's a bit of redundancy in the cleanUncommittedAppends logic, so did a refactoring there. Also cleaned up the preconditions check for format 3 and 4 for delete files, since it's the same condition, we can collapse those cases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Cleaning up uncommitted cleanup logic in SnapshotProducer implementations [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao merged PR #14923:
URL: https://github.com/apache/iceberg/pull/14923


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Small cleanup in MergingSnapshotProducer cleanUncommittedAppends [iceberg]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #14923:
URL: https://github.com/apache/iceberg/pull/14923#discussion_r2644200626


##########
core/src/main/java/org/apache/iceberg/SnapshotProducer.java:
##########
@@ -654,6 +654,22 @@ protected List<ManifestFile> writeDataManifests(
     return writeManifests(files, group -> writeDataFileGroup(group, dataSeq, spec));
   }
 
+  // Deletes uncommitted manifests; clears list if clearManifests and any deleted.
+  protected void deleteUncommitted(
+      List<ManifestFile> manifests, Set<ManifestFile> committed, boolean clearManifests) {
+    boolean anyDeleted = false;

Review Comment:
   While we are here, I believe the [cleanUncommitted](https://github.com/apache/iceberg/blob/f25e07db6318184272d5c98dede4d268ee5288ab/core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java#L347) is basically doing the same thing, with a different signature. It may be worth handling this case too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Small cleanup in MergingSnapshotProducer cleanUncommittedAppends [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #14923:
URL: https://github.com/apache/iceberg/pull/14923#discussion_r2644278680


##########
core/src/main/java/org/apache/iceberg/SnapshotProducer.java:
##########
@@ -654,6 +654,22 @@ protected List<ManifestFile> writeDataManifests(
     return writeManifests(files, group -> writeDataFileGroup(group, dataSeq, spec));
   }
 
+  // Deletes uncommitted manifests; clears list if clearManifests and any deleted.
+  protected void deleteUncommitted(
+      List<ManifestFile> manifests, Set<ManifestFile> committed, boolean clearManifests) {
+    boolean anyDeleted = false;

Review Comment:
   Cleaned this up as well! 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Small cleanup in MergingSnapshotProducer cleanUncommittedAppends [iceberg]

Posted by "dramaticlly (via GitHub)" <gi...@apache.org>.
dramaticlly commented on code in PR #14923:
URL: https://github.com/apache/iceberg/pull/14923#discussion_r2643947733


##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -991,60 +985,42 @@ public Object updateEvent() {
     return new CreateSnapshotEvent(tableName, operation(), snapshotId, sequenceNumber, summary);
   }
 
-  @SuppressWarnings("checkstyle:CyclomaticComplexity")
-  private void cleanUncommittedAppends(Set<ManifestFile> committed) {
-    if (!cachedNewDataManifests.isEmpty()) {
-      boolean hasDeletes = false;
-      for (ManifestFile manifest : cachedNewDataManifests) {
-        if (!committed.contains(manifest)) {
-          deleteFile(manifest.path());
-          hasDeletes = true;
-        }
-      }
-
-      if (hasDeletes) {
-        this.cachedNewDataManifests.clear();
-      }
-    }
-
-    boolean hasDeleteDeletes = false;
-    for (ManifestFile cachedNewDeleteManifest : cachedNewDeleteManifests) {
-      if (!committed.contains(cachedNewDeleteManifest)) {
-        deleteFile(cachedNewDeleteManifest.path());
-        hasDeleteDeletes = true;
-      }
-    }
-
-    if (hasDeleteDeletes) {
-      this.cachedNewDeleteManifests.clear();
-    }
+  @Override
+  protected void cleanUncommitted(Set<ManifestFile> committed) {
+    mergeManager.cleanUncommitted(committed);
+    filterManager.cleanUncommitted(committed);
+    deleteMergeManager.cleanUncommitted(committed);
+    deleteFilterManager.cleanUncommitted(committed);
+    cleanUncommittedAppends(committed);
+  }
 
+  private void cleanUncommittedAppends(Set<ManifestFile> committed) {
+    deleteUncommitted(cachedNewDataManifests, committed, true /* clear manifests */);
+    deleteUncommitted(cachedNewDeleteManifests, committed, true /* clear manifests */);
     // rewritten manifests are always owned by the table
-    for (ManifestFile manifest : rewrittenAppendManifests) {
-      if (!committed.contains(manifest)) {
-        deleteFile(manifest.path());
-      }
-    }
+    deleteUncommitted(rewrittenAppendManifests, committed, false);
 
     // manifests that are not rewritten are only owned by the table if the commit succeeded
     if (!committed.isEmpty()) {
       // the commit succeeded if at least one manifest was committed
       // the table now owns appendManifests; clean up any that are not used
-      for (ManifestFile manifest : appendManifests) {
-        if (!committed.contains(manifest)) {
-          deleteFile(manifest.path());
-        }
-      }
+      deleteUncommitted(appendManifests, committed, false);
     }
   }
 
-  @Override
-  protected void cleanUncommitted(Set<ManifestFile> committed) {
-    mergeManager.cleanUncommitted(committed);
-    filterManager.cleanUncommitted(committed);
-    deleteMergeManager.cleanUncommitted(committed);
-    deleteFilterManager.cleanUncommitted(committed);
-    cleanUncommittedAppends(committed);
+  private void deleteUncommitted(

Review Comment:
   this function looks stateless, I am wondering if we can benefit the same cleanup in FastAppend (https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/FastAppend.java#L184-L194) by pulling this class into SnapshotProducer instead?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Small cleanup in MergingSnapshotProducer cleanUncommittedAppends [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #14923:
URL: https://github.com/apache/iceberg/pull/14923#discussion_r2644162150


##########
core/src/main/java/org/apache/iceberg/MergingSnapshotProducer.java:
##########
@@ -991,60 +985,42 @@ public Object updateEvent() {
     return new CreateSnapshotEvent(tableName, operation(), snapshotId, sequenceNumber, summary);
   }
 
-  @SuppressWarnings("checkstyle:CyclomaticComplexity")
-  private void cleanUncommittedAppends(Set<ManifestFile> committed) {
-    if (!cachedNewDataManifests.isEmpty()) {
-      boolean hasDeletes = false;
-      for (ManifestFile manifest : cachedNewDataManifests) {
-        if (!committed.contains(manifest)) {
-          deleteFile(manifest.path());
-          hasDeletes = true;
-        }
-      }
-
-      if (hasDeletes) {
-        this.cachedNewDataManifests.clear();
-      }
-    }
-
-    boolean hasDeleteDeletes = false;
-    for (ManifestFile cachedNewDeleteManifest : cachedNewDeleteManifests) {
-      if (!committed.contains(cachedNewDeleteManifest)) {
-        deleteFile(cachedNewDeleteManifest.path());
-        hasDeleteDeletes = true;
-      }
-    }
-
-    if (hasDeleteDeletes) {
-      this.cachedNewDeleteManifests.clear();
-    }
+  @Override
+  protected void cleanUncommitted(Set<ManifestFile> committed) {
+    mergeManager.cleanUncommitted(committed);
+    filterManager.cleanUncommitted(committed);
+    deleteMergeManager.cleanUncommitted(committed);
+    deleteFilterManager.cleanUncommitted(committed);
+    cleanUncommittedAppends(committed);
+  }
 
+  private void cleanUncommittedAppends(Set<ManifestFile> committed) {
+    deleteUncommitted(cachedNewDataManifests, committed, true /* clear manifests */);
+    deleteUncommitted(cachedNewDeleteManifests, committed, true /* clear manifests */);
     // rewritten manifests are always owned by the table
-    for (ManifestFile manifest : rewrittenAppendManifests) {
-      if (!committed.contains(manifest)) {
-        deleteFile(manifest.path());
-      }
-    }
+    deleteUncommitted(rewrittenAppendManifests, committed, false);
 
     // manifests that are not rewritten are only owned by the table if the commit succeeded
     if (!committed.isEmpty()) {
       // the commit succeeded if at least one manifest was committed
       // the table now owns appendManifests; clean up any that are not used
-      for (ManifestFile manifest : appendManifests) {
-        if (!committed.contains(manifest)) {
-          deleteFile(manifest.path());
-        }
-      }
+      deleteUncommitted(appendManifests, committed, false);
     }
   }
 
-  @Override
-  protected void cleanUncommitted(Set<ManifestFile> committed) {
-    mergeManager.cleanUncommitted(committed);
-    filterManager.cleanUncommitted(committed);
-    deleteMergeManager.cleanUncommitted(committed);
-    deleteFilterManager.cleanUncommitted(committed);
-    cleanUncommittedAppends(committed);
+  private void deleteUncommitted(

Review Comment:
   great find, yeah I think we can further clean this up!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Cleaning up uncommitted cleanup logic in SnapshotProducer implementations [iceberg]

Posted by "huaxingao (via GitHub)" <gi...@apache.org>.
huaxingao commented on PR #14923:
URL: https://github.com/apache/iceberg/pull/14923#issuecomment-3688547612

   Thanks @amogh-jahagirdar for the PR! Thanks @dramaticlly @geruh @singhpk234 for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Small cleanup in MergingSnapshotProducer cleanUncommittedAppends [iceberg]

Posted by "geruh (via GitHub)" <gi...@apache.org>.
geruh commented on code in PR #14923:
URL: https://github.com/apache/iceberg/pull/14923#discussion_r2644200626


##########
core/src/main/java/org/apache/iceberg/SnapshotProducer.java:
##########
@@ -654,6 +654,22 @@ protected List<ManifestFile> writeDataManifests(
     return writeManifests(files, group -> writeDataFileGroup(group, dataSeq, spec));
   }
 
+  // Deletes uncommitted manifests; clears list if clearManifests and any deleted.
+  protected void deleteUncommitted(
+      List<ManifestFile> manifests, Set<ManifestFile> committed, boolean clearManifests) {
+    boolean anyDeleted = false;

Review Comment:
   While we are here, I believe the [cleanUncommitted](https://github.com/apache/iceberg/blob/f25e07db6318184272d5c98dede4d268ee5288ab/core/src/main/java/org/apache/iceberg/BaseRewriteManifests.java#L347) is basically doing the same thing, with a different signature. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org