[jira] [Created] (HADOOP-17981) Support etag-assisted renames in FileOutputCommitter

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-17981:
---------------------------------------

             Summary: Support etag-assisted renames in FileOutputCommitter
                 Key: HADOOP-17981
                 URL: https://issues.apache.org/jira/browse/HADOOP-17981
             Project: Hadoop Common
          Issue Type: New Feature
          Components: fs, fs/azure
    Affects Versions: 3.4.0
            Reporter: Steve Loughran
            Assignee: Steve Loughran


To deal with some throttling/retry issues in object stores,
pass the FileStatus entries retrieved during listing
into a private interface ResilientCommitByRename which filesystems
may implement to use extra attributes in the listing (etag, version)
to constrain and validate the operation.

Although targeting azure, GCS and others could use. no point in S3A as they shouldn't use this committer.

# And we are not going to do any changes to FileSystem as there are explicit guarantees of public use and stability.
I am not going to make that change as the hive thing that will suddenly start expecting it to work forever.
# I'm not planning to merge this in, as the manifest committer is going to include this and more (MAPREDUCE-7341)

However, I do need to get this in on a branch, so am doing this work on trunk for dev & test and for others to review



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org