[GitHub] [iceberg] dramaticlly commented on a diff in pull request #6889: Spark 3.3: Skip duplicate check on deleted file path when import files

["dramaticlly (via GitHub)" <gi...@apache.org>]

dramaticlly commented on code in PR #6889:
URL: https://github.com/apache/iceberg/pull/6889#discussion_r1112349858


##########
spark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java:
##########
@@ -605,7 +605,8 @@ public static void importSparkPartitions(
           filesToImport
               .map((MapFunction<DataFile, String>) f -> f.path().toString(), Encoders.STRING())
               .toDF("file_path");
-      Dataset<Row> existingFiles = loadMetadataTable(spark, targetTable, MetadataTableType.ENTRIES);
+      Dataset<Row> entriesFiles = loadMetadataTable(spark, targetTable, MetadataTableType.ENTRIES);
+      Dataset<Row> existingFiles = entriesFiles.filter(entriesFiles.col("status").notEqual(2));

Review Comment:
   I wanted to use `ManifestEntry.Status.DELETED` instead but looks like the class is not public per https://github.com/apache/iceberg/blob/master/core/src/main/java/org/apache/iceberg/ManifestEntry.java#L31



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org