[GitHub] [iceberg] aokolnychyi commented on pull request #1313: Move RemoveOrphanFilesAction run in Driver to Excutor

[GitBox <gi...@apache.org>]

aokolnychyi commented on pull request #1313:
URL: https://github.com/apache/iceberg/pull/1313#issuecomment-671514522


   I would like to make sure I get the use case correctly. Is the main purpose to use `RemoveOrphanFilesAction` to clean up expired files? If so, I think that's a misuse of the API. Instead, one should use `ExpireSnapshotsAction` that Russell is working on in PR #1264.
   
   The most expensive part in `RemoveOrphanFilesAction` is listing. The bigger table the more pressure you put on the storage. The number of orphan files should be usually small so the delete should be quick. This action should be called maybe once a month or so. `ExpireSnapshotsAction`, on the other hand, is meant to be called regularly. It will have an executor service to parallelize deletes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org