答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk

[Yang Weiwei <ch...@hotmail.com>]

Hi Steve


The code is available in HDFS-7240 feature branch, public git repo here<https://github.com/apache/hadoop/tree/HDFS-7240>.

I am not sure if there is a "public" API for object stores, but the design doc<https://issues.apache.org/jira/secure/attachment/12799549/ozone_user_v0.pdf> uses most common syntax so I believe it should be compliance. You can find the rest API doc here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneRest.md> (with some example usages), and commandline API here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneCommandShell.md>.


Look forward for your feedback!


--Weiwei


________________________________
发件人: Steve Loughran <st...@hortonworks.com>
发送时间: 2017年10月20日 11:49
收件人: Yang Weiwei
抄送: hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; common-dev@hadoop.apache.org
主题: Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk


Wow, big piece of work

1. Where is a PR/branch on github with rendered docs for us to look at?
2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.

thanks

> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
>  1.  All services, management scripts
>  2.  Object store APIs, exposed via both REST and RPC
>  3.  Master service UIs, command line interfaces
>  4.  Pluggable pipeline Integration
>  5.  Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
>  6.  Corona - a load generator for Ozone.
>  7.  Essential documentation added to Hadoop site.
>  8.  Version specific Ozone Documentation, accessible via service UI.
>  9.  Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
Dev cluster with docker - Hadoop - Apache Software Foundation<https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker>
cwiki.apache.org
First, it uses a much more smaller common image which doesn't contains Hadoop. Second, the real Hadoop should be built from the source and the dist director should be ...



>
>
> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang