[jira] [Commented] (HADOOP-13998) initial s3guard preview

["Aaron Fabbri (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849666#comment-15849666 ] 

Aaron Fabbri commented on HADOOP-13998:
---------------------------------------

All of the dependencies for this have patches available for review.  I would like to start on empty directory handling improvements, but would prefer to merge the feature branch to trunk first to avoid having to maintain more S3AFileSystem diffs.

*I'm proposing we merge* HADOOP-13345 to trunk as soon as we get the dependent JIRAs linked here committed.  I'll provide a summary of where we are at below.  I look forward to feedback from [~stevel@apache.org], [~cnauroth], [~eddyxu], [~mackrorysd], and the rest of the community.

The main feature we want for the initial version is listing consistency, and we've accomplished that.

For testing, we have completed (off the top of my head):
- List consistency tests with failure injection.  (HADOOP-13793) This integration test forces a delay in visibility of certain files by wrapping the AWS S3 client. It asserts listing is consistent.  The test fails without S3Guard, and succeeds with it. 
- All existing S3 integration tests with and without S3Guard.  The filesystem contract tests have been invaluable here. (HADOOP-13589 makes these very easy to run).
- MetadataStore contract tests that ensure that the API semantics of the DynamoDB and in-memory reference implementations are correct.
- MetadataStore scale tests that can be used to force DynamoDB service throttling and ensure we are robust to that.
- Unit tests for different parts of the S3Guard logic.

In addition to this upstream testing, my colleagues have run a couple of our in-house test harnesses against S3Guard.  This includes Hive, Spark, and a number of other components.  All the testing is looking great so far.


> initial s3guard preview
> -----------------------
>
>                 Key: HADOOP-13998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13998
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Steve Loughran
>
> JIRA to link in all the things we think are needed for a preview/merge into trunk



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org