You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2016/11/02 18:16:58 UTC
[jira] [Created] (HADOOP-13786) add output committer which uses
s3guard for consistent O(1) commits to S3
Steve Loughran created HADOOP-13786:
---------------------------------------
Summary: add output committer which uses s3guard for consistent O(1) commits to S3
Key: HADOOP-13786
URL: https://issues.apache.org/jira/browse/HADOOP-13786
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.0.0-alpha2
Reporter: Steve Loughran
Assignee: Steve Loughran
A goal of this code is "support O(1) commits to S3 repositories in the presence of failures". Implement it, including whatever is needed to demonstrate the correctness of the algorithm. (that is, assuming that s3guard provides a consistent view of the presence/absence of blobs, show that we can commit directly).
I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie. not visible until the close()), if we need to use that to allow us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org