You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/03/02 23:46:44 UTC
[GitHub] [incubator-iceberg] rdblue commented on issue #822: File leaking in
RemoveSnapshots API
rdblue commented on issue #822: File leaking in RemoveSnapshots API
URL: https://github.com/apache/incubator-iceberg/issues/822#issuecomment-593681753
I'm re-posting my reply from the dev list here:
The section of code that you point out is _mostly_ correct, but there is a case that we should fix. The reason I say "mostly correct" is that it has correct behavior and has error handling to ensure that as much of the work is done as possible.
Even if not all of the data is cleaned up, the behavior must be to throw an exception if and only if the metadata commit was successful. If the commit is successful, then throwing an error puts a client in an unrecoverable state because it thinks that the table hasn't changed when it actually did. If a client retries an operation that was already committed successfully, it could make incorrect changes to the table.
To address reliability, cleanup uses `Tasks` so that [a single failure doesn't prevent attempting to delete other files](https://github.com/apache/incubator-iceberg/blob/master/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java#L281) and files that can't be deleted are logged. There is similar failure handling while finding files to delete, so that [read failures don't prevent deleting](https://github.com/apache/incubator-iceberg/blob/master/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java#L291) as much data as possible. If you want even more guarantees, then you can add a custom delete method and persist the files to delete for another service to handle.
Now that I'm looking at this, one area that can be improved is the handling of manifest list files. There is [no failure handling while reading manifest lists](https://github.com/apache/incubator-iceberg/blob/master/core/src/main/java/org/apache/iceberg/RemoveSnapshots.java#L333-L344), so we should add that.
I should also point out that this is not the only way for data to leak. For example, an executor failure in Spark can lead to a data file getting written by not committed to a table. I recommend having a way to periodically list table locations and check against the metadata tables, like `all_data_files` and `all_manifests`, to ensure that no files are left behind.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org