[GitHub] [iceberg] szehon-ho commented on a change in pull request #4370: Core: Make removal of delete files optional in manifest filtering

[GitBox <gi...@apache.org>]

szehon-ho commented on a change in pull request #4370:
URL: https://github.com/apache/iceberg/pull/4370#discussion_r831631088



##########
File path: core/src/main/java/org/apache/iceberg/ManifestFilterManager.java
##########
@@ -360,23 +360,23 @@ private boolean manifestHasDeletedFiles(PartitionAndMetricsEvaluator evaluator,
           dropPartitions.contains(file.specId(), file.partition()) ||
           (isDelete && entry.sequenceNumber() > 0 && entry.sequenceNumber() < minSequenceNumber);
 
-      boolean nonMatchingDeleteFile = !file.content().equals(FileContent.DATA) && !evaluator.rowsMustMatch(file);
-      if (!markedForDelete && nonMatchingDeleteFile) {
-        // not all DeleteFiles removal can be handled by metadata operation, skip in this case
-        continue;
-      }
-
       if (markedForDelete || evaluator.rowsMightMatch(file)) {
+        boolean allRowsMatch = markedForDelete || evaluator.rowsMustMatch(file);
         ValidationException.check(
-            markedForDelete || evaluator.rowsMustMatch(file),
+            allRowsMatch || isDelete, // ignore delete files where some records may not match the expression
             "Cannot delete file where some, but not all, rows match filter %s: %s",
             this.deleteExpression, file.path());
 
-        hasDeletedFiles = true;
+        hasDeletedFiles = allRowsMatch;
+
         if (failAnyDelete) {
           throw new DeleteException(reader.spec().partitionToPath(file.partition()));

Review comment:
       In failAnyDelete case, this would  trigger if we have a deleteFile but rows do not match.  Probably this never happens in delete call, but wonder if we need to update the condition (failAnyDelete && allRowsMatch).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org