You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/10/01 22:42:30 UTC
[GitHub] [iceberg] jackye1995 opened a new issue #3212: Extend DeleteOrphanFiles use cases
jackye1995 opened a new issue #3212:
URL: https://github.com/apache/iceberg/issues/3212
`DeleteOrphanFiles` currently has the following limitations:
1. it assume a single location for all the files, which would not work for tables that stores files in a separated directory
2. it does not work if multiple tables store files under the same root, which would be a common case for people having object storage mode to store all data files at the root location
3. it still requires a file system to perform file listing, so for S3FileIO users they still have to install S3A file system just to remove orphan files
My current thoughts are the following:
For 1, can be mitigated by running the action multiple times with different root locations, which would probably be enough as a maintenance procedure.
For 2, maybe we can add a few filters, which includes:
1. path filter: allowing a regex to know if a path might belong to some other table and should not be removed
2. metadata filter: allowing checks against some specific metadata of a file to know if it might belong to some other table. This also requires adding a new `Map<String, String> metadata()` interface in `InputFile` for `FileIO` to implement.
3. we also have the hard-coded restriction that only lists `at most 3 levels and only dirs that have less than 10 direct sub dirs on the driver`, which I think should be made configurable.
For 3, I am not sure what is the best way. I am thinking if we could add listing operation also to `FileIO`, and it would only be used for this action. Would listing be useful for any other potential actions? Or would it cause a potential abuse of file listing?
@aokolnychyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #3212: Extend DeleteOrphanFiles use cases
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #3212:
URL: https://github.com/apache/iceberg/issues/3212#issuecomment-971807958
For clarification, in (2), by `object storage mode`, do you mean the [`ObjectStorageLocationProvider`](https://github.com/apache/iceberg/blob/12e30ebaeb6dc0ff7e6bb01c0d9f8ae8a5f06ddf/core/src/main/java/org/apache/iceberg/LocationProviders.java#L98)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on issue #3212: Extend DeleteOrphanFiles use cases
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on issue #3212:
URL: https://github.com/apache/iceberg/issues/3212#issuecomment-971815162
> For clarification, in (2), by object storage mode, do you mean the ObjectStorageLocationProvider?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on issue #3212: Extend DeleteOrphanFiles use cases
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on issue #3212:
URL: https://github.com/apache/iceberg/issues/3212#issuecomment-934989975
Yea I was also thinking about this issue.
For (2.1) and (2.2), maybe we can have a multi-table Action instead of having user pass in path and metadata filter which could be dangerous. And yes for (2.3), I was thinking this for awhile.
For (3) I think yes we could add FileIO.list and be vigilant in the reviews to not allow listing unless absolutely necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org