[GitHub] [incubator-iceberg] mehtaashish23 commented on issue #758: Add retry framework to Hadoop table load

[GitBox <gi...@apache.org>]

mehtaashish23 commented on issue #758: Add retry framework to Hadoop table load
URL: https://github.com/apache/incubator-iceberg/issues/758#issuecomment-579903297
 
 
   Guys, correct me if I am wrong, I feel this doesn't look like problem with rename guarantee, its about guarantees while reading the `version-hint.text` file during read time operation (according to above stack trace) in a environment where there are many writers trying to update the table, in sequence or parallel. 
   
   IIUC after doing the `rename` from `GUID.metadata.json` to `version.metadata.json`, we try to use `create with overwrite` [here] to update the version-hint.text with the current version number. The behavior of this API, is to update the content of this file in place, and during that time, if someone is reading this file might see different behavior, based on underlying filesystem guarantees i.e the file doesn't exist, or the data in the file is momentarily empty etc. 
   
   I suppose, we would need retries in read path of versionHint. *Also* if above is true, then we need better documentation, around guarantees we expect w.r.t `create with overwrite` from underlying filesystem.
   
   [here]: https://github.com/apache/incubator-iceberg/blob/master/core/src/main/java/org/apache/iceberg/hadoop/HadoopTableOperations.java#L220

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org