[jira] [Commented] (HADOOP-13786) add output committer which uses s3guard for consistent O(1) commits to S3

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715389#comment-15715389 ] 

Steve Loughran commented on HADOOP-13786:
-----------------------------------------

Update: the way s3a does commits via mergeOrUpdate() isn't really that atomic. It's atomic per destination rename, but as it treewalks doing a merge, there are actual potentials for races/inconsistent outcomes. What s3guard is at least guarantee that the view of the FS during that walk is consistent across clients.

I'm checking in/pushing up an update, where I've been slowly moving up to some s3a internals, more for performance than consistency.

Examples:

* switch to {{innerDelete(FileStatus)}} for the delete, adding a parameter there to allow the check for maybe creating a parent dir to be skipped. As we know there's about be a new child, skipping that avoids 1-3 HEAD calls and a PUT which will soon be deleted
* switch to an implementation of {{FileSystem.rename/3}}. This raises exceptions and lets us choose overwrite policy (good), but it still does two getFileStatusCalls, one for the src and one for the dest. As we have the source and have just deleted the test, we don't need them again. Having an @Privat method which took the source and dest values would strip out another 2-6 HTTP requests. Admittedly, on a large commit the cost of his preamble is low —the COPY becomes the expense.


> add output committer which uses s3guard for consistent O(1) commits to S3
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-13786
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13786
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>
> A goal of this code is "support O(1) commits to S3 repositories in the presence of failures". Implement it, including whatever is needed to demonstrate the correctness of the algorithm. (that is, assuming that s3guard provides a consistent view of the presence/absence of blobs, show that we can commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output streams (ie. not visible until the close()), if we need to use that to allow us to abort commit operations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org