[jira] [Commented] (HADOOP-13998) initial s3guard preview

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032962#comment-16032962 ] 

Steve Loughran commented on HADOOP-13998:
-----------------------------------------

it's time to merge in, as the branch is big enough we should merge in trunk to the branch again, have everything working nicely; the troublesome SSE-C test can be skipped and covered in docs/release notes.

Before then, last chance to do refactorings and renamings of options, etc, and review.

h3. Code
Need to look at code to see if
# perf on non-s3guard codepaths is impacted, e.g. by new requests
# config options look good before freezing their names
# all those places which have TODO in them: are they critical? If not, do they at least have JIRA coverage somewhere

h3. Docs
# docs: how well do they read for someone who hasn't worked on it (this can evolve, obviously)
# release notes. Maybe: make clear it's experimental, safe in non-auth, auth is more dangerous

h3. Testing!

what functional tests have people done?

I'm trying to do some downstream testing in my [spark cloud integration module|https://github.com/hortonworks-spark/cloud-integration] project, mostly on committer. But I've just moved the spark SQL Hive test suite {{org.apache.spark.sql.sources.HadoopFsRelationTest}} & its ORC subclass in so I can verify that the commit algorithms work there. I'm now trying to switch over to the inconsistent client to see if I can make it easy to observe inconsistencies in the classic "legacy" file commit algorithms. (ignoring Parquet for technical reasons; that'll need a fix in spark itself).

Other logistics
# get some review from [~cnauroth] if he has the time
# submit a single patch to cover the merge

> initial s3guard preview
> -----------------------
>
>                 Key: HADOOP-13998
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13998
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Steve Loughran
>
> JIRA to link in all the things we think are needed for a preview/merge into trunk



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org