[GitHub] [iceberg] rdblue commented on issue #1398: No retries on snapshot commit on eventual consistent file system

[GitBox <gi...@apache.org>]

rdblue commented on issue #1398:
URL: https://github.com/apache/iceberg/issues/1398#issuecomment-682837392


   Our S3 file system doesn't do the `getFileStatus` check when overwriting, which is how we avoid this. I think the best way to handle this is to add a certain amount of retries at the `FileIO` layer.
   
   We already have retries for `table.refresh()` (in `BaseMetastoreTableOperations`) that take care of most of these issues for the root metadata. The reason why we added the retries there was to avoid adding extra latency when there actually is a problem -- when a data file is missing, for example. But, it looks like it would be better to have retries for at least metadata files when using S3.
   
   I think it makes sense to add retries to `HadoopInputFile` and to make the number of retries and total retry timeout configurable either in table properties or Hadoop Configuration. The downside to adding retries there is that we'd have retries for data files as well, but that seems like a reasonable trade-off to me.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org