[GitHub] [hadoop] steveloughran opened a new pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy

[GitBox <gi...@apache.org>]

steveloughran opened a new pull request #1229: HADOOP-16490. Improve S3Guard handling of FNFEs in copy
URL: https://github.com/apache/hadoop/pull/1229
 
 
   S3Guard retry policy independently configurable from other retry policies,
   
   * and use a setting with exponential backoff
   * new config names
   * copy raises a RemoteFileChangedException which is *not* caught in rename() and downgraded to false. Thus: when a rename is unrecoverable, this fact is propagated
   * tests for this
   
   Also: tests turning auth mode on/off have to handle the auth state being set through an authoritative path, rather than a single flag. Caught me out as of course the first test I saw with this was the ITestS3ARemoteFileChanged rename ones, and I assumed that it was my new code. It was actually due to me setting an auth path last week.
   
   I'm unsure about the use of "RemoteFileChangedException", but it could be a remote file change, especially for an etag, where the file has been deleted since being recorded.I don't know if/how we should adapt to that in S3Guard if the problem persists. The file is either *no longer there* or *changes not yet visible*. Maybe consider marking parent dir as nonauth? Logging error, always?
   
   Change-Id: I7bb468aca0f4019537d82bc083f0a9887eaa282b

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org