You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/03/17 08:02:24 UTC

[GitHub] [iceberg] flyrain commented on issue #4346: Make DeleteOrphanFiles in Spark reliable

flyrain commented on issue #4346:
URL: https://github.com/apache/iceberg/issues/4346#issuecomment-1070508331


   Not sure if we've already used crc file. I'd think it can increase the comparison reliability. Here is the reason:
   1. We can NOT tell two files are identical if their crc values are the same due to the high collision rate of crc. But we can tell two files are different if their crc are different, which is still useful. For example, we've known the crc values of valid file a, b, c under a dir. We then have the confidence to delete any files without crc or without a valid crc value.
   2. Each file generated by Spark has its crc file, so we don't have to calculate them on the fly. We can even preload them in manifest file if needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org