You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "amogh-jahagirdar (via GitHub)" <gi...@apache.org> on 2023/12/05 15:30:36 UTC

[PR] Core: Fix logic for determining set of committed files [iceberg]

amogh-jahagirdar opened a new pull request, #9221:
URL: https://github.com/apache/iceberg/pull/9221

   This is a follow up to https://github.com/apache/iceberg/pull/9183. Null gets returned from `committedFiles` in 2 cases:
   
   1.)  The snapshot ID list is empty. This snapshot ID is the set of snapshots produced as a result of the transaction
   
   2.).The set of committed files could not exactly be determined (e.g. missing any snapshot metadata).
   
   In the first case, the logic should perform cleanup in case there were no new snapshots (e.g. ExpireSnapshots) which #9183 pointed out and addressed.
   
   In the second case, we do want to signal to callers that it's a unique situation and file deletions should not be performed.
   
   What this change does is return an empty set in case 1 and null in case 2 to prevent deleting files when we could not determine the full set of committed files.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix logic for determining set of committed files [iceberg]

Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #9221:
URL: https://github.com/apache/iceberg/pull/9221#issuecomment-1841035245

   https://github.com/apache/iceberg/pull/9183 @bartash  already added tests for the ExpireSnapshots case, but based on the tests passing before that implies there weren't any tests for case 2 above (since those would have caught the behavior change in deletion in case snapshots couldn't be looked up). It may be difficult to test for that case (have to force the lookup of a committed snapshot to be null right after the transaction commits) , but I'll look into it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix logic in BaseTransaction for determining set of committed files when there are no new snapshots [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue commented on code in PR #9221:
URL: https://github.com/apache/iceberg/pull/9221#discussion_r1416089227


##########
core/src/main/java/org/apache/iceberg/BaseTransaction.java:
##########
@@ -446,16 +447,18 @@ private void commitSimpleTransaction() {
       }
 
       Set<String> committedFiles = committedFiles(ops, newSnapshots);
-      // delete all of the files that were deleted in the most recent set of operation commits
-      Tasks.foreach(deletedFiles)
-          .suppressFailureWhenFinished()
-          .onFailure((file, exc) -> LOG.warn("Failed to delete uncommitted file: {}", file, exc))
-          .run(
-              path -> {
-                if (committedFiles == null || !committedFiles.contains(path)) {
-                  ops.io().deleteFile(path);
-                }
-              });
+      if (committedFiles != null) {
+        // delete all of the files that were deleted in the most recent set of operation commits
+        Tasks.foreach(deletedFiles)
+            .suppressFailureWhenFinished()
+            .onFailure((file, exc) -> LOG.warn("Failed to delete uncommitted file: {}", file, exc))
+            .run(
+                path -> {
+                  if (!committedFiles.contains(path)) {
+                    ops.io().deleteFile(path);
+                  }
+                });
+      }

Review Comment:
   We need to restore the warning.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Core: Fix logic in BaseTransaction for determining set of committed files when there are no new snapshots [iceberg]

Posted by "rdblue (via GitHub)" <gi...@apache.org>.
rdblue merged PR #9221:
URL: https://github.com/apache/iceberg/pull/9221


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org