You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/19 22:37:20 UTC
[GitHub] [iceberg] karuppayya commented on a diff in pull request #4652: ICEBERG-4346: Better handling of Orphan files
karuppayya commented on code in PR #4652:
URL: https://github.com/apache/iceberg/pull/4652#discussion_r925010212
##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -94,15 +100,10 @@
extends BaseSparkAction<DeleteOrphanFilesSparkAction> implements DeleteOrphanFiles {
private static final Logger LOG = LoggerFactory.getLogger(DeleteOrphanFilesSparkAction.class);
- private static final UserDefinedFunction filenameUDF = functions.udf((String path) -> {
- int lastIndex = path.lastIndexOf(File.separator);
- if (lastIndex == -1) {
- return path;
- } else {
- return path.substring(lastIndex + 1);
- }
- }, DataTypes.StringType);
+ private PrefixMismatchMode prefixMismatchMode = PrefixMismatchMode.ERROR;
Review Comment:
done
##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -143,6 +144,24 @@ public DeleteOrphanFilesSparkAction executeDeleteWith(ExecutorService executorSe
return this;
}
+ @Override
+ public DeleteOrphanFiles prefixMismatchMode(PrefixMismatchMode mismatchMode) {
Review Comment:
done
##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -143,6 +144,24 @@ public DeleteOrphanFilesSparkAction executeDeleteWith(ExecutorService executorSe
return this;
}
+ @Override
+ public DeleteOrphanFiles prefixMismatchMode(PrefixMismatchMode mismatchMode) {
+ this.prefixMismatchMode = mismatchMode;
+ return this;
+ }
+
+ @Override
+ public DeleteOrphanFiles equalSchemes(Map<String, String> schemes) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org