You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/03/14 15:47:11 UTC
[GitHub] [iceberg] ulmako commented on pull request #4307: WIP: Spark 3.2: Allow disabling HiddenPathFilter in RemoveOrphansFiles
ulmako commented on pull request #4307:
URL: https://github.com/apache/iceberg/pull/4307#issuecomment-1066976791
Hey @rdblue,
thanks for all your feedback.
The formatting stuff got mixed up before I set up the code-style scheme and I did not check it. I will obviously revert those changes.
Your suggestion to use the `options` in `BaseSparkAction` is definitely the better choice.
Regarding the automatic detection, by looking into the partition field names, I have one question: would you still ignore all other hidden paths, that are not related to the partitions or disable the HiddenPathFilter completely. For illustration purposes:
Given the partition name `_part` and the following paths:
```
-- /data/
| -- _part=AA
| -- _part=BB
| -- _part=CC
| -- _some-folder
```
Would you only include the paths `_part=*` and ignore the path `_some-folder` or include all paths?
Also, where should I document the new option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org