[GitHub] [iceberg] xloya commented on a change in pull request #3724: [Spark][Core]: Expire data files that read 0 pieces of data during rewriting

[GitBox <gi...@apache.org>]

xloya commented on a change in pull request #3724:
URL: https://github.com/apache/iceberg/pull/3724#discussion_r768258460



##########
File path: core/src/main/java/org/apache/iceberg/actions/RewriteDataFilesCommitManager.java
##########
@@ -79,11 +83,34 @@ public void commitFileGroups(Set<RewriteFileGroup> fileGroups) {
     }
 
     RewriteFiles rewrite = table.newRewrite().validateFromSnapshot(startingSnapshotId);
-    if (useStartingSequenceNumber) {
-      long sequenceNumber = table.snapshot(startingSnapshotId).sequenceNumber();
-      rewrite.rewriteFiles(rewrittenDataFiles, addedDataFiles, sequenceNumber);
+
+    if (addedDataFiles.size() > 0) {

Review comment:
       My initial idea was to rewrite these data files into an empty file or delete the restrictions here, but I'm not sure whether it affects other logic. If it can be removed, I think this is the easiest.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org