You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Yang Weiwei <ch...@hotmail.com> on 2017/10/19 01:54:44 UTC
[DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Hello everyone,
I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
The merge payload includes the following:
1. All services, management scripts
2. Object store APIs, exposed via both REST and RPC
3. Master service UIs, command line interfaces
4. Pluggable pipeline Integration
5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
6. Corona - a load generator for Ozone.
7. Essential documentation added to Hadoop site.
8. Version specific Ozone Documentation, accessible via service UI.
9. Docker support for ozone, which enables faster development cycles.
To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
Please share your thoughts, thanks!
-- Weiwei Yang
答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Yang Weiwei <ch...@hotmail.com>.
Hi Steve
The code is available in HDFS-7240 feature branch, public git repo here<https://github.com/apache/hadoop/tree/HDFS-7240>.
I am not sure if there is a "public" API for object stores, but the design doc<https://issues.apache.org/jira/secure/attachment/12799549/ozone_user_v0.pdf> uses most common syntax so I believe it should be compliance. You can find the rest API doc here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneRest.md> (with some example usages), and commandline API here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneCommandShell.md>.
Look forward for your feedback!
--Weiwei
________________________________
发件人: Steve Loughran <st...@hortonworks.com>
发送时间: 2017年10月20日 11:49
收件人: Yang Weiwei
抄送: hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; common-dev@hadoop.apache.org
主题: Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Wow, big piece of work
1. Where is a PR/branch on github with rendered docs for us to look at?
2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.
thanks
> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
Dev cluster with docker - Hadoop - Apache Software Foundation<https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker>
cwiki.apache.org
First, it uses a much more smaller common image which doesn't contains Hadoop. Second, the real Hadoop should be built from the source and the dist director should be ...
>
>
> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Yang Weiwei <ch...@hotmail.com>.
Hi Steve
The code is available in HDFS-7240 feature branch, public git repo here<https://github.com/apache/hadoop/tree/HDFS-7240>.
I am not sure if there is a "public" API for object stores, but the design doc<https://issues.apache.org/jira/secure/attachment/12799549/ozone_user_v0.pdf> uses most common syntax so I believe it should be compliance. You can find the rest API doc here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneRest.md> (with some example usages), and commandline API here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneCommandShell.md>.
Look forward for your feedback!
--Weiwei
________________________________
发件人: Steve Loughran <st...@hortonworks.com>
发送时间: 2017年10月20日 11:49
收件人: Yang Weiwei
抄送: hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; common-dev@hadoop.apache.org
主题: Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Wow, big piece of work
1. Where is a PR/branch on github with rendered docs for us to look at?
2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.
thanks
> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
Dev cluster with docker - Hadoop - Apache Software Foundation<https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker>
cwiki.apache.org
First, it uses a much more smaller common image which doesn't contains Hadoop. Second, the real Hadoop should be built from the source and the dist director should be ...
>
>
> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Yang Weiwei <ch...@hotmail.com>.
Hi Steve
The code is available in HDFS-7240 feature branch, public git repo here<https://github.com/apache/hadoop/tree/HDFS-7240>.
I am not sure if there is a "public" API for object stores, but the design doc<https://issues.apache.org/jira/secure/attachment/12799549/ozone_user_v0.pdf> uses most common syntax so I believe it should be compliance. You can find the rest API doc here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneRest.md> (with some example usages), and commandline API here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneCommandShell.md>.
Look forward for your feedback!
--Weiwei
________________________________
发件人: Steve Loughran <st...@hortonworks.com>
发送时间: 2017年10月20日 11:49
收件人: Yang Weiwei
抄送: hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; common-dev@hadoop.apache.org
主题: Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Wow, big piece of work
1. Where is a PR/branch on github with rendered docs for us to look at?
2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.
thanks
> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
Dev cluster with docker - Hadoop - Apache Software Foundation<https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker>
cwiki.apache.org
First, it uses a much more smaller common image which doesn't contains Hadoop. Second, the real Hadoop should be built from the source and the dist director should be ...
>
>
> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
答复: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Yang Weiwei <ch...@hotmail.com>.
Hi Steve
The code is available in HDFS-7240 feature branch, public git repo here<https://github.com/apache/hadoop/tree/HDFS-7240>.
I am not sure if there is a "public" API for object stores, but the design doc<https://issues.apache.org/jira/secure/attachment/12799549/ozone_user_v0.pdf> uses most common syntax so I believe it should be compliance. You can find the rest API doc here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneRest.md> (with some example usages), and commandline API here<https://github.com/apache/hadoop/blob/HDFS-7240/hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/OzoneCommandShell.md>.
Look forward for your feedback!
--Weiwei
________________________________
发件人: Steve Loughran <st...@hortonworks.com>
发送时间: 2017年10月20日 11:49
收件人: Yang Weiwei
抄送: hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; common-dev@hadoop.apache.org
主题: Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Wow, big piece of work
1. Where is a PR/branch on github with rendered docs for us to look at?
2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.
thanks
> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
Dev cluster with docker - Hadoop - Apache Software Foundation<https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker>
cwiki.apache.org
First, it uses a much more smaller common image which doesn't contains Hadoop. Second, the real Hadoop should be built from the source and the dist director should be ...
>
>
> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Larry McCay <lm...@hortonworks.com>.
For such sizable merges in Hadoop, I would like to start doing security audits in order to have an initial idea of the attack surface, the protections available for known threats, what sort of configuration is being used to launch processes, etc.
I dug into the architecture documents while in the middle of this list - nice docs!
I do intend to try and make a generic check list like this for such security audits in the future so a lot of this is from that but I tried to also direct specific questions from those docs as well.
1. UIs
I see there are at least two UIs - Storage Container Manager and Key Space Manager. There are a number of typical vulnerabilities that we find in UIs
1.1. What sort of validation is being done on any accepted user input? (pointers to code would be appreciated)
1.2. What explicit protections have been built in for (pointers to code would be appreciated):
1.2.1. cross site scripting
1.2.2. cross site request forgery
1.2.3. click jacking (X-Frame-Options)
1.3. What sort of authentication is required for access to the UIs?
1.4. What authorization is available for determining who can access what capabilities of the UIs for either viewing, modifying data or affecting object stores and related processes?
1.5. Are the UIs built with proxying in mind by leveraging X-Forwarded headers?
1.6. Is there any input that will ultimately be persisted in configuration for executing shell commands or processes?
1.7. Do the UIs support the trusted proxy pattern with doas impersonation?
1.8. Is there TLS/SSL support?
2. REST APIs
2.1. Do the REST APIs support the trusted proxy pattern with doas impersonation capabilities?
2.2. What explicit protections have been built in for:
2.2.1. cross site scripting (XSS)
2.2.2. cross site request forgery (CSRF)
2.2.3. XML External Entity (XXE)
2.3. What is being used for authentication - Hadoop Auth Module?
2.4. Are there separate processes for the HTTP resources (UIs and REST endpoints) or are the part of existing HDFS processes?
2.5. Is there TLS/SSL support?
2.6. Are there new CLI commands and/or clients for access the REST APIs?
2.7. Bucket Level API allows for setting of ACLs on a bucket - what authorization is required here - is there a restrictive ACL set on creation?
2.8. Bucket Level API allows for deleting a bucket - I assume this is dependent on ACLs based access control?
2.9. Bucket Level API to list bucket returns up to 1000 keys - is there paging available?
2.10. Storage Level APIs indicate “Signed with User Authorization” what does this refer to exactly?
2.11. Object Level APIs indicate that there is no ACL support and only bucket owners can read and write - but there are ACL APIs on the Bucket Level are they meaningless for now?
2.12. How does a REST client know which Ozone Handler to connect to or am I missing some well known NN type endpoint in the architecture doc somewhere?
3. Encryption
3.1. Is there any support for encryption of persisted data?
3.2. If so, is KMS and the hadoop key command used for key management?
4. Configuration
4.1. Are there any passwords or secrets being added to configuration?
4.2. If so, are they accessed via Configuration.getPassword() to allow for provisioning in credential providers?
4.3. Are there any settings that are used to launch docker containers or shell out any commands, etc?
5. HA
5.1. Are there provisions for HA?
5.2. Are we leveraging the existing HA capabilities in HDFS?
5.3. Is Storage Container Manager a SPOF?
5.4. I see HA listed in future work in the architecture doc - is this still an open issue?
> On Oct 20, 2017, at 7:49 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
>
> Wow, big piece of work
>
> 1. Where is a PR/branch on github with rendered docs for us to look at?
> 2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.
>
> thanks
>
>> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>>
>> Hello everyone,
>>
>>
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>>
>>
>>
>> The merge payload includes the following:
>>
>> 1. All services, management scripts
>> 2. Object store APIs, exposed via both REST and RPC
>> 3. Master service UIs, command line interfaces
>> 4. Pluggable pipeline Integration
>> 5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
>> 6. Corona - a load generator for Ozone.
>> 7. Essential documentation added to Hadoop site.
>> 8. Version specific Ozone Documentation, accessible via service UI.
>> 9. Docker support for ozone, which enables faster development cycles.
>>
>>
>> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
>>
>>
>> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>>
>>
>> Please share your thoughts, thanks!
>>
>>
>> -- Weiwei Yang
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Steve Loughran <st...@hortonworks.com>.
Wow, big piece of work
1. Where is a PR/branch on github with rendered docs for us to look at?
2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.
thanks
> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi Sanjay,
With respect to Ozone my two main concerns were:
1. Wether Ozone can help scaling out the namespace service in handling
higher RPC workloads.
I think we came to common conclusion that using Ozone as a block management
layer is a reasonable path to scaling HDFS.
The discussions are in-progress and probably will go on for a while.
2. Releasing Hadoop with components that don't have security. I was asking
for a design doc at a minimum.
I think the document Anu published addresses it.
Thanks,
Konstantin
On Tue, Feb 20, 2018 at 3:06 PM, sanjay Radia <sa...@gmail.com> wrote:
>
>
>
> Konstantine
>
>
> Thanks for your feedback and comments over the last few months. Have we
> addressed all your issues and concerns?
>
> sanjay
>
>
> > On Feb 13, 2018, at 6:28 PM, sanjay Radia <sa...@gmail.com> wrote:
> >
> > Sorry the formatting got messed by my email client. Here it is again
> >
> >
> > Dear
> > Hadoop Community Members,
> >
> > We had multiple community discussions, a few meetings in smaller groups
> and also jira discussions with respect to this thread. We express our
> gratitude for participation and valuable comments.
> >
> > The key questions raised were following
> > 1) How the new block storage layer and OzoneFS benefit HDFS and we were
> asked to chalk out a roadmap towards the goal of a scalable namenode
> working with the new storage layer
> > 2) We were asked to provide a security design
> > 3)There were questions around stability given ozone brings in a large
> body of code.
> > 4) Why can’t they be separate projects forever or merged in when
> production ready?
> >
> > We have responded to all the above questions with detailed explanations
> and answers on the jira as well as in the discussions. We believe that
> should sufficiently address community’s concerns.
> >
> > Please see the summary below:
> >
> > 1) The new code base benefits HDFS scaling and a roadmap has been
> provided.
> >
> > Summary:
> > - New block storage layer addresses the scalability of the block layer.
> We have shown how existing NN can be connected to the new block layer and
> its benefits. We have shown 2 milestones, 1st milestone is much simpler
> than 2nd milestone while giving almost the same scaling benefits.
> Originally we had proposed simply milestone 2 and the community felt that
> removing the FSN/BM lock was was a fair amount of work and a simpler
> solution would be useful
> > - We provide a new K-V namespace called Ozone FS with
> FileSystem/FileContext plugins to allow the users to use the new system.
> BTW Hive and Spark work very well on KV-namespaces on the cloud. This will
> facilitate stabilizing the new block layer.
> > - The new block layer has a new netty based protocol engine in the
> Datanode which, when stabilized, can be used by the old hdfs block layer.
> See details below on sharing of code.
> >
> >
> > 2) Stability impact on the existing HDFS code base and code separation.
> The new block layer and the OzoneFS are in modules that are separate from
> old HDFS code - currently there are no calls from HDFS into Ozone except
> for DN starting the new block layer module if configured to do so. It does
> not add instability (the instability argument has been raised many times).
> Over time as we share code, we will ensure that the old HDFS continues to
> remains stable. (for example we plan to stabilize the new netty based
> protocol engine in the new block layer before sharing it with HDFS’s old
> block layer)
> >
> >
> > 3) In the short term and medium term, the new system and HDFS will be
> used side-by-side by users. Side by-side usage in the short term for
> testing and side-by-side in the medium term for actual production use till
> the new system has feature parity with old HDFS. During this time, sharing
> the DN daemon and admin functions between the two systems is operationally
> important:
> > - Sharing DN daemon to avoid additional operational daemon lifecycle
> management
> > - Common decommissioning of the daemon and DN: One place to decommission
> for a node and its storage.
> > - Replacing failed disks and internal balancing capacity across disks -
> this needs to be done for both the current HDFS blocks and the new
> block-layer blocks.
> > - Balancer: we would like use the same balancer and provide a common way
> to balance and common management of the bandwidth used for balancing
> > - Security configuration setup - reuse existing set up for DNs rather
> then a new one for an independent cluster.
> >
> >
> > 4) Need to easily share the block layer code between the two systems
> when used side-by-side. Areas where sharing code is desired over time:
> > - Sharing new block layer’s new netty based protocol engine for old
> HDFS DNs (a long time sore issue for HDFS block layer).
> > - Shallow data copy from old system to new system is practical only if
> within same project and daemon otherwise have to deal with security setting
> and coordinations across daemons. Shallow copy is useful as customer
> migrate from old to new.
> > - Shared disk scheduling in the future and in the short term have a
> single round robin rather than independent round robins.
> > While sharing code across projects is technically possible (anything is
> possible in software), it is significantly harder typically requiring
> cleaner public apis etc. Sharing within a project though internal APIs is
> often simpler (such as the protocol engine that we want to share).
> >
> >
> > 5) Security design, including a threat model and and the solution has
> been posted.
> > 6) Temporary Separation and merge later: Several of the comments in the
> jira have argued that we temporarily separate the two code bases for now
> and then later merge them when the new code is stable:
> >
> > - If there is agreement to merge later, why bother separating now -
> there needs to be to be good reasons to separate now. We have addressed
> the stability and separation of the new code from existing above.
> > - Merge the new code back into HDFS later will be harder.
> >
> > **The code and goals will diverge further.
> > ** We will be taking on extra work to split and then take extra work
> to merge.
> > ** The issues raised today will be raised all the same then.
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi Sanjay,
With respect to Ozone my two main concerns were:
1. Wether Ozone can help scaling out the namespace service in handling
higher RPC workloads.
I think we came to common conclusion that using Ozone as a block management
layer is a reasonable path to scaling HDFS.
The discussions are in-progress and probably will go on for a while.
2. Releasing Hadoop with components that don't have security. I was asking
for a design doc at a minimum.
I think the document Anu published addresses it.
Thanks,
Konstantin
On Tue, Feb 20, 2018 at 3:06 PM, sanjay Radia <sa...@gmail.com> wrote:
>
>
>
> Konstantine
>
>
> Thanks for your feedback and comments over the last few months. Have we
> addressed all your issues and concerns?
>
> sanjay
>
>
> > On Feb 13, 2018, at 6:28 PM, sanjay Radia <sa...@gmail.com> wrote:
> >
> > Sorry the formatting got messed by my email client. Here it is again
> >
> >
> > Dear
> > Hadoop Community Members,
> >
> > We had multiple community discussions, a few meetings in smaller groups
> and also jira discussions with respect to this thread. We express our
> gratitude for participation and valuable comments.
> >
> > The key questions raised were following
> > 1) How the new block storage layer and OzoneFS benefit HDFS and we were
> asked to chalk out a roadmap towards the goal of a scalable namenode
> working with the new storage layer
> > 2) We were asked to provide a security design
> > 3)There were questions around stability given ozone brings in a large
> body of code.
> > 4) Why can’t they be separate projects forever or merged in when
> production ready?
> >
> > We have responded to all the above questions with detailed explanations
> and answers on the jira as well as in the discussions. We believe that
> should sufficiently address community’s concerns.
> >
> > Please see the summary below:
> >
> > 1) The new code base benefits HDFS scaling and a roadmap has been
> provided.
> >
> > Summary:
> > - New block storage layer addresses the scalability of the block layer.
> We have shown how existing NN can be connected to the new block layer and
> its benefits. We have shown 2 milestones, 1st milestone is much simpler
> than 2nd milestone while giving almost the same scaling benefits.
> Originally we had proposed simply milestone 2 and the community felt that
> removing the FSN/BM lock was was a fair amount of work and a simpler
> solution would be useful
> > - We provide a new K-V namespace called Ozone FS with
> FileSystem/FileContext plugins to allow the users to use the new system.
> BTW Hive and Spark work very well on KV-namespaces on the cloud. This will
> facilitate stabilizing the new block layer.
> > - The new block layer has a new netty based protocol engine in the
> Datanode which, when stabilized, can be used by the old hdfs block layer.
> See details below on sharing of code.
> >
> >
> > 2) Stability impact on the existing HDFS code base and code separation.
> The new block layer and the OzoneFS are in modules that are separate from
> old HDFS code - currently there are no calls from HDFS into Ozone except
> for DN starting the new block layer module if configured to do so. It does
> not add instability (the instability argument has been raised many times).
> Over time as we share code, we will ensure that the old HDFS continues to
> remains stable. (for example we plan to stabilize the new netty based
> protocol engine in the new block layer before sharing it with HDFS’s old
> block layer)
> >
> >
> > 3) In the short term and medium term, the new system and HDFS will be
> used side-by-side by users. Side by-side usage in the short term for
> testing and side-by-side in the medium term for actual production use till
> the new system has feature parity with old HDFS. During this time, sharing
> the DN daemon and admin functions between the two systems is operationally
> important:
> > - Sharing DN daemon to avoid additional operational daemon lifecycle
> management
> > - Common decommissioning of the daemon and DN: One place to decommission
> for a node and its storage.
> > - Replacing failed disks and internal balancing capacity across disks -
> this needs to be done for both the current HDFS blocks and the new
> block-layer blocks.
> > - Balancer: we would like use the same balancer and provide a common way
> to balance and common management of the bandwidth used for balancing
> > - Security configuration setup - reuse existing set up for DNs rather
> then a new one for an independent cluster.
> >
> >
> > 4) Need to easily share the block layer code between the two systems
> when used side-by-side. Areas where sharing code is desired over time:
> > - Sharing new block layer’s new netty based protocol engine for old
> HDFS DNs (a long time sore issue for HDFS block layer).
> > - Shallow data copy from old system to new system is practical only if
> within same project and daemon otherwise have to deal with security setting
> and coordinations across daemons. Shallow copy is useful as customer
> migrate from old to new.
> > - Shared disk scheduling in the future and in the short term have a
> single round robin rather than independent round robins.
> > While sharing code across projects is technically possible (anything is
> possible in software), it is significantly harder typically requiring
> cleaner public apis etc. Sharing within a project though internal APIs is
> often simpler (such as the protocol engine that we want to share).
> >
> >
> > 5) Security design, including a threat model and and the solution has
> been posted.
> > 6) Temporary Separation and merge later: Several of the comments in the
> jira have argued that we temporarily separate the two code bases for now
> and then later merge them when the new code is stable:
> >
> > - If there is agreement to merge later, why bother separating now -
> there needs to be to be good reasons to separate now. We have addressed
> the stability and separation of the new code from existing above.
> > - Merge the new code back into HDFS later will be harder.
> >
> > **The code and goals will diverge further.
> > ** We will be taking on extra work to split and then take extra work
> to merge.
> > ** The issues raised today will be raised all the same then.
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi Sanjay,
With respect to Ozone my two main concerns were:
1. Wether Ozone can help scaling out the namespace service in handling
higher RPC workloads.
I think we came to common conclusion that using Ozone as a block management
layer is a reasonable path to scaling HDFS.
The discussions are in-progress and probably will go on for a while.
2. Releasing Hadoop with components that don't have security. I was asking
for a design doc at a minimum.
I think the document Anu published addresses it.
Thanks,
Konstantin
On Tue, Feb 20, 2018 at 3:06 PM, sanjay Radia <sa...@gmail.com> wrote:
>
>
>
> Konstantine
>
>
> Thanks for your feedback and comments over the last few months. Have we
> addressed all your issues and concerns?
>
> sanjay
>
>
> > On Feb 13, 2018, at 6:28 PM, sanjay Radia <sa...@gmail.com> wrote:
> >
> > Sorry the formatting got messed by my email client. Here it is again
> >
> >
> > Dear
> > Hadoop Community Members,
> >
> > We had multiple community discussions, a few meetings in smaller groups
> and also jira discussions with respect to this thread. We express our
> gratitude for participation and valuable comments.
> >
> > The key questions raised were following
> > 1) How the new block storage layer and OzoneFS benefit HDFS and we were
> asked to chalk out a roadmap towards the goal of a scalable namenode
> working with the new storage layer
> > 2) We were asked to provide a security design
> > 3)There were questions around stability given ozone brings in a large
> body of code.
> > 4) Why can’t they be separate projects forever or merged in when
> production ready?
> >
> > We have responded to all the above questions with detailed explanations
> and answers on the jira as well as in the discussions. We believe that
> should sufficiently address community’s concerns.
> >
> > Please see the summary below:
> >
> > 1) The new code base benefits HDFS scaling and a roadmap has been
> provided.
> >
> > Summary:
> > - New block storage layer addresses the scalability of the block layer.
> We have shown how existing NN can be connected to the new block layer and
> its benefits. We have shown 2 milestones, 1st milestone is much simpler
> than 2nd milestone while giving almost the same scaling benefits.
> Originally we had proposed simply milestone 2 and the community felt that
> removing the FSN/BM lock was was a fair amount of work and a simpler
> solution would be useful
> > - We provide a new K-V namespace called Ozone FS with
> FileSystem/FileContext plugins to allow the users to use the new system.
> BTW Hive and Spark work very well on KV-namespaces on the cloud. This will
> facilitate stabilizing the new block layer.
> > - The new block layer has a new netty based protocol engine in the
> Datanode which, when stabilized, can be used by the old hdfs block layer.
> See details below on sharing of code.
> >
> >
> > 2) Stability impact on the existing HDFS code base and code separation.
> The new block layer and the OzoneFS are in modules that are separate from
> old HDFS code - currently there are no calls from HDFS into Ozone except
> for DN starting the new block layer module if configured to do so. It does
> not add instability (the instability argument has been raised many times).
> Over time as we share code, we will ensure that the old HDFS continues to
> remains stable. (for example we plan to stabilize the new netty based
> protocol engine in the new block layer before sharing it with HDFS’s old
> block layer)
> >
> >
> > 3) In the short term and medium term, the new system and HDFS will be
> used side-by-side by users. Side by-side usage in the short term for
> testing and side-by-side in the medium term for actual production use till
> the new system has feature parity with old HDFS. During this time, sharing
> the DN daemon and admin functions between the two systems is operationally
> important:
> > - Sharing DN daemon to avoid additional operational daemon lifecycle
> management
> > - Common decommissioning of the daemon and DN: One place to decommission
> for a node and its storage.
> > - Replacing failed disks and internal balancing capacity across disks -
> this needs to be done for both the current HDFS blocks and the new
> block-layer blocks.
> > - Balancer: we would like use the same balancer and provide a common way
> to balance and common management of the bandwidth used for balancing
> > - Security configuration setup - reuse existing set up for DNs rather
> then a new one for an independent cluster.
> >
> >
> > 4) Need to easily share the block layer code between the two systems
> when used side-by-side. Areas where sharing code is desired over time:
> > - Sharing new block layer’s new netty based protocol engine for old
> HDFS DNs (a long time sore issue for HDFS block layer).
> > - Shallow data copy from old system to new system is practical only if
> within same project and daemon otherwise have to deal with security setting
> and coordinations across daemons. Shallow copy is useful as customer
> migrate from old to new.
> > - Shared disk scheduling in the future and in the short term have a
> single round robin rather than independent round robins.
> > While sharing code across projects is technically possible (anything is
> possible in software), it is significantly harder typically requiring
> cleaner public apis etc. Sharing within a project though internal APIs is
> often simpler (such as the protocol engine that we want to share).
> >
> >
> > 5) Security design, including a threat model and and the solution has
> been posted.
> > 6) Temporary Separation and merge later: Several of the comments in the
> jira have argued that we temporarily separate the two code bases for now
> and then later merge them when the new code is stable:
> >
> > - If there is agreement to merge later, why bother separating now -
> there needs to be to be good reasons to separate now. We have addressed
> the stability and separation of the new code from existing above.
> > - Merge the new code back into HDFS later will be harder.
> >
> > **The code and goals will diverge further.
> > ** We will be taking on extra work to split and then take extra work
> to merge.
> > ** The issues raised today will be raised all the same then.
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi Sanjay,
With respect to Ozone my two main concerns were:
1. Wether Ozone can help scaling out the namespace service in handling
higher RPC workloads.
I think we came to common conclusion that using Ozone as a block management
layer is a reasonable path to scaling HDFS.
The discussions are in-progress and probably will go on for a while.
2. Releasing Hadoop with components that don't have security. I was asking
for a design doc at a minimum.
I think the document Anu published addresses it.
Thanks,
Konstantin
On Tue, Feb 20, 2018 at 3:06 PM, sanjay Radia <sa...@gmail.com> wrote:
>
>
>
> Konstantine
>
>
> Thanks for your feedback and comments over the last few months. Have we
> addressed all your issues and concerns?
>
> sanjay
>
>
> > On Feb 13, 2018, at 6:28 PM, sanjay Radia <sa...@gmail.com> wrote:
> >
> > Sorry the formatting got messed by my email client. Here it is again
> >
> >
> > Dear
> > Hadoop Community Members,
> >
> > We had multiple community discussions, a few meetings in smaller groups
> and also jira discussions with respect to this thread. We express our
> gratitude for participation and valuable comments.
> >
> > The key questions raised were following
> > 1) How the new block storage layer and OzoneFS benefit HDFS and we were
> asked to chalk out a roadmap towards the goal of a scalable namenode
> working with the new storage layer
> > 2) We were asked to provide a security design
> > 3)There were questions around stability given ozone brings in a large
> body of code.
> > 4) Why can’t they be separate projects forever or merged in when
> production ready?
> >
> > We have responded to all the above questions with detailed explanations
> and answers on the jira as well as in the discussions. We believe that
> should sufficiently address community’s concerns.
> >
> > Please see the summary below:
> >
> > 1) The new code base benefits HDFS scaling and a roadmap has been
> provided.
> >
> > Summary:
> > - New block storage layer addresses the scalability of the block layer.
> We have shown how existing NN can be connected to the new block layer and
> its benefits. We have shown 2 milestones, 1st milestone is much simpler
> than 2nd milestone while giving almost the same scaling benefits.
> Originally we had proposed simply milestone 2 and the community felt that
> removing the FSN/BM lock was was a fair amount of work and a simpler
> solution would be useful
> > - We provide a new K-V namespace called Ozone FS with
> FileSystem/FileContext plugins to allow the users to use the new system.
> BTW Hive and Spark work very well on KV-namespaces on the cloud. This will
> facilitate stabilizing the new block layer.
> > - The new block layer has a new netty based protocol engine in the
> Datanode which, when stabilized, can be used by the old hdfs block layer.
> See details below on sharing of code.
> >
> >
> > 2) Stability impact on the existing HDFS code base and code separation.
> The new block layer and the OzoneFS are in modules that are separate from
> old HDFS code - currently there are no calls from HDFS into Ozone except
> for DN starting the new block layer module if configured to do so. It does
> not add instability (the instability argument has been raised many times).
> Over time as we share code, we will ensure that the old HDFS continues to
> remains stable. (for example we plan to stabilize the new netty based
> protocol engine in the new block layer before sharing it with HDFS’s old
> block layer)
> >
> >
> > 3) In the short term and medium term, the new system and HDFS will be
> used side-by-side by users. Side by-side usage in the short term for
> testing and side-by-side in the medium term for actual production use till
> the new system has feature parity with old HDFS. During this time, sharing
> the DN daemon and admin functions between the two systems is operationally
> important:
> > - Sharing DN daemon to avoid additional operational daemon lifecycle
> management
> > - Common decommissioning of the daemon and DN: One place to decommission
> for a node and its storage.
> > - Replacing failed disks and internal balancing capacity across disks -
> this needs to be done for both the current HDFS blocks and the new
> block-layer blocks.
> > - Balancer: we would like use the same balancer and provide a common way
> to balance and common management of the bandwidth used for balancing
> > - Security configuration setup - reuse existing set up for DNs rather
> then a new one for an independent cluster.
> >
> >
> > 4) Need to easily share the block layer code between the two systems
> when used side-by-side. Areas where sharing code is desired over time:
> > - Sharing new block layer’s new netty based protocol engine for old
> HDFS DNs (a long time sore issue for HDFS block layer).
> > - Shallow data copy from old system to new system is practical only if
> within same project and daemon otherwise have to deal with security setting
> and coordinations across daemons. Shallow copy is useful as customer
> migrate from old to new.
> > - Shared disk scheduling in the future and in the short term have a
> single round robin rather than independent round robins.
> > While sharing code across projects is technically possible (anything is
> possible in software), it is significantly harder typically requiring
> cleaner public apis etc. Sharing within a project though internal APIs is
> often simpler (such as the protocol engine that we want to share).
> >
> >
> > 5) Security design, including a threat model and and the solution has
> been posted.
> > 6) Temporary Separation and merge later: Several of the comments in the
> jira have argued that we temporarily separate the two code bases for now
> and then later merge them when the new code is stable:
> >
> > - If there is agreement to merge later, why bother separating now -
> there needs to be to be good reasons to separate now. We have addressed
> the stability and separation of the new code from existing above.
> > - Merge the new code back into HDFS later will be harder.
> >
> > **The code and goals will diverge further.
> > ** We will be taking on extra work to split and then take extra work
> to merge.
> > ** The issues raised today will be raised all the same then.
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Konstantine
Thanks for your feedback and comments over the last few months. Have we addressed all your issues and concerns?
sanjay
> On Feb 13, 2018, at 6:28 PM, sanjay Radia <sa...@gmail.com> wrote:
>
> Sorry the formatting got messed by my email client. Here it is again
>
>
> Dear
> Hadoop Community Members,
>
> We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments.
>
> The key questions raised were following
> 1) How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
> 2) We were asked to provide a security design
> 3)There were questions around stability given ozone brings in a large body of code.
> 4) Why can’t they be separate projects forever or merged in when production ready?
>
> We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns.
>
> Please see the summary below:
>
> 1) The new code base benefits HDFS scaling and a roadmap has been provided.
>
> Summary:
> - New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful
> - We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
> - The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
>
>
> 2) Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
>
>
> 3) In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
> - Sharing DN daemon to avoid additional operational daemon lifecycle management
> - Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
> - Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
> - Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
> - Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
>
>
> 4) Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
> - Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
> - Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
> - Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
> While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
>
>
> 5) Security design, including a threat model and and the solution has been posted.
> 6) Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
>
> - If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
> - Merge the new code back into HDFS later will be harder.
>
> **The code and goals will diverge further.
> ** We will be taking on extra work to split and then take extra work to merge.
> ** The issues raised today will be raised all the same then.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Konstantine
Thanks for your feedback and comments over the last few months. Have we addressed all your issues and concerns?
sanjay
> On Feb 13, 2018, at 6:28 PM, sanjay Radia <sa...@gmail.com> wrote:
>
> Sorry the formatting got messed by my email client. Here it is again
>
>
> Dear
> Hadoop Community Members,
>
> We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments.
>
> The key questions raised were following
> 1) How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
> 2) We were asked to provide a security design
> 3)There were questions around stability given ozone brings in a large body of code.
> 4) Why can’t they be separate projects forever or merged in when production ready?
>
> We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns.
>
> Please see the summary below:
>
> 1) The new code base benefits HDFS scaling and a roadmap has been provided.
>
> Summary:
> - New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful
> - We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
> - The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
>
>
> 2) Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
>
>
> 3) In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
> - Sharing DN daemon to avoid additional operational daemon lifecycle management
> - Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
> - Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
> - Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
> - Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
>
>
> 4) Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
> - Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
> - Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
> - Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
> While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
>
>
> 5) Security design, including a threat model and and the solution has been posted.
> 6) Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
>
> - If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
> - Merge the new code back into HDFS later will be harder.
>
> **The code and goals will diverge further.
> ** We will be taking on extra work to split and then take extra work to merge.
> ** The issues raised today will be raised all the same then.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Konstantine
Thanks for your feedback and comments over the last few months. Have we addressed all your issues and concerns?
sanjay
> On Feb 13, 2018, at 6:28 PM, sanjay Radia <sa...@gmail.com> wrote:
>
> Sorry the formatting got messed by my email client. Here it is again
>
>
> Dear
> Hadoop Community Members,
>
> We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments.
>
> The key questions raised were following
> 1) How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
> 2) We were asked to provide a security design
> 3)There were questions around stability given ozone brings in a large body of code.
> 4) Why can’t they be separate projects forever or merged in when production ready?
>
> We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns.
>
> Please see the summary below:
>
> 1) The new code base benefits HDFS scaling and a roadmap has been provided.
>
> Summary:
> - New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful
> - We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
> - The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
>
>
> 2) Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
>
>
> 3) In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
> - Sharing DN daemon to avoid additional operational daemon lifecycle management
> - Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
> - Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
> - Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
> - Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
>
>
> 4) Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
> - Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
> - Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
> - Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
> While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
>
>
> 5) Security design, including a threat model and and the solution has been posted.
> 6) Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
>
> - If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
> - Merge the new code back into HDFS later will be harder.
>
> **The code and goals will diverge further.
> ** We will be taking on extra work to split and then take extra work to merge.
> ** The issues raised today will be raised all the same then.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Konstantine
Thanks for your feedback and comments over the last few months. Have we addressed all your issues and concerns?
sanjay
> On Feb 13, 2018, at 6:28 PM, sanjay Radia <sa...@gmail.com> wrote:
>
> Sorry the formatting got messed by my email client. Here it is again
>
>
> Dear
> Hadoop Community Members,
>
> We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments.
>
> The key questions raised were following
> 1) How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
> 2) We were asked to provide a security design
> 3)There were questions around stability given ozone brings in a large body of code.
> 4) Why can’t they be separate projects forever or merged in when production ready?
>
> We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns.
>
> Please see the summary below:
>
> 1) The new code base benefits HDFS scaling and a roadmap has been provided.
>
> Summary:
> - New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful
> - We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
> - The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
>
>
> 2) Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
>
>
> 3) In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
> - Sharing DN daemon to avoid additional operational daemon lifecycle management
> - Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
> - Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
> - Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
> - Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
>
>
> 4) Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
> - Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
> - Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
> - Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
> While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
>
>
> 5) Security design, including a threat model and and the solution has been posted.
> 6) Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
>
> - If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
> - Merge the new code back into HDFS later will be harder.
>
> **The code and goals will diverge further.
> ** We will be taking on extra work to split and then take extra work to merge.
> ** The issues raised today will be raised all the same then.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Sorry the formatting got messed by my email client. Here it is again
Dear
Hadoop Community Members,
We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments.
The key questions raised were following
1) How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
2) We were asked to provide a security design
3)There were questions around stability given ozone brings in a large body of code.
4) Why can’t they be separate projects forever or merged in when production ready?
We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns.
Please see the summary below:
1) The new code base benefits HDFS scaling and a roadmap has been provided.
Summary:
- New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful
- We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
- The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
2) Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
3) In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
- Sharing DN daemon to avoid additional operational daemon lifecycle management
- Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
- Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
- Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
- Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
4) Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
- Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
- Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
- Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
5) Security design, including a threat model and and the solution has been posted.
6) Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
- If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
- Merge the new code back into HDFS later will be harder.
**The code and goals will diverge further.
** We will be taking on extra work to split and then take extra work to merge.
** The issues raised today will be raised all the same then.
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Sorry the formatting got messed by my email client. Here it is again
Dear
Hadoop Community Members,
We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments.
The key questions raised were following
1) How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
2) We were asked to provide a security design
3)There were questions around stability given ozone brings in a large body of code.
4) Why can’t they be separate projects forever or merged in when production ready?
We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns.
Please see the summary below:
1) The new code base benefits HDFS scaling and a roadmap has been provided.
Summary:
- New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful
- We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
- The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
2) Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
3) In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
- Sharing DN daemon to avoid additional operational daemon lifecycle management
- Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
- Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
- Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
- Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
4) Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
- Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
- Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
- Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
5) Security design, including a threat model and and the solution has been posted.
6) Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
- If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
- Merge the new code back into HDFS later will be harder.
**The code and goals will diverge further.
** We will be taking on extra work to split and then take extra work to merge.
** The issues raised today will be raised all the same then.
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Sorry the formatting got messed by my email client. Here it is again
Dear
Hadoop Community Members,
We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments.
The key questions raised were following
1) How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
2) We were asked to provide a security design
3)There were questions around stability given ozone brings in a large body of code.
4) Why can’t they be separate projects forever or merged in when production ready?
We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns.
Please see the summary below:
1) The new code base benefits HDFS scaling and a roadmap has been provided.
Summary:
- New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful
- We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
- The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
2) Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
3) In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
- Sharing DN daemon to avoid additional operational daemon lifecycle management
- Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
- Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
- Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
- Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
4) Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
- Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
- Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
- Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
5) Security design, including a threat model and and the solution has been posted.
6) Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
- If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
- Merge the new code back into HDFS later will be harder.
**The code and goals will diverge further.
** We will be taking on extra work to split and then take extra work to merge.
** The issues raised today will be raised all the same then.
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Sorry the formatting got messed by my email client. Here it is again
Dear
Hadoop Community Members,
We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments.
The key questions raised were following
1) How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
2) We were asked to provide a security design
3)There were questions around stability given ozone brings in a large body of code.
4) Why can’t they be separate projects forever or merged in when production ready?
We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns.
Please see the summary below:
1) The new code base benefits HDFS scaling and a roadmap has been provided.
Summary:
- New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful
- We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
- The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
2) Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
3) In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
- Sharing DN daemon to avoid additional operational daemon lifecycle management
- Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
- Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
- Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
- Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
4) Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
- Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
- Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
- Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
5) Security design, including a threat model and and the solution has been posted.
6) Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
- If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
- Merge the new code back into HDFS later will be harder.
**The code and goals will diverge further.
** We will be taking on extra work to split and then take extra work to merge.
** The issues raised today will be raised all the same then.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Dear Hadoop Community Members,
We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments. The key questions raised were following
How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
We were asked to provide a security design
There were questions around stability given ozone brings in a large body of code.
Why can’t they be separate projects forever or merged in when production ready?
We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns. Please see the summary below:
The new code base benefits to HDFS scaling and a roadmap has been provided. Summary:
New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful.
We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
Sharing DN daemon to avoid additional operational daemon lifecycle management
Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
Security design, including a threat model and and the solution has been posted.
Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
Merge the new code back into HDFS later will be harder.
The code and goals will diverge further.
We will be taking on extra work to split and then take extra work to merge.
The issues raised today will be raised all the same then.
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Dear Hadoop Community Members,
We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments. The key questions raised were following
How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
We were asked to provide a security design
There were questions around stability given ozone brings in a large body of code.
Why can’t they be separate projects forever or merged in when production ready?
We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns. Please see the summary below:
The new code base benefits to HDFS scaling and a roadmap has been provided. Summary:
New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful.
We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
Sharing DN daemon to avoid additional operational daemon lifecycle management
Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
Security design, including a threat model and and the solution has been posted.
Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
Merge the new code back into HDFS later will be harder.
The code and goals will diverge further.
We will be taking on extra work to split and then take extra work to merge.
The issues raised today will be raised all the same then.
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Steve Loughran <st...@hortonworks.com>.
Wow, big piece of work
1. Where is a PR/branch on github with rendered docs for us to look at?
2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.
thanks
> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Jitendra Pandey <ji...@hortonworks.com>.
Hi Konstantin,
Thank you for taking out time to review ozone. I appreciate your comments and questions.
> There are two main limitations in HDFS
> a) The throughput of Namespace operations. Which is limited by the
> number of RPCs the NameNode can handle
> b) The number of objects (files + blocks) the system can maintain.
> Which is limited by the memory size of the NameNode.
I agree completely. We believe ozone attempts to address both these issues for HDFS.
Let us look at the Number of objects problem. Ozone directly addresses the scalability of number of blocks by introducing storage containers that can hold multiple blocks together. The earlier efforts on this were complicated by the fact that block manager and namespace are intertwined in HDFS Namenode. There have been efforts in past to separate block manager from namespace for e.g. HDFS-5477. Ozone addresses this problem by cleanly separating the block layer. Separation of block layer also addresses the file/directories scalability because it frees up the blockmap from the namenode.
Separate block layer relieves namenode from handling block reports, IBRs, heartbeats, replication monitor etc, and thus reduces the contention on FSNamesystem lock and significantly reduces the GC pressure on the namenode. These improvements will greatly help the RPC performance of the Namenode.
> Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118. The design doc for the new architecture has never been
> published.
We do believe that Namenode can leverage the ozone’s storage container layer, however, that is also a big effort. We would like to first have block layer stabilized in ozone before taking that up. However, we would certainly support any community effort on that, and in fact it was brought up in last BoF session at the summit.
Big data is evolving rapidly. We see our customers needing scalable file systems, Objects stores(like S3) and Block Store(for docker and VMs). Ozone improves HDFS in two ways. It addresses throughput and scale issues of HDFS, and enriches it with newer capabilities.
> Ozone is a big enough system to deserve its own project.
I took a quick look at the core code in ozone and the cloc command reports 22,511 lines of functionality changes in Java.
This patch also brings in web framework code like Angular.js and that brings in bunch of css and js files that contribute to the size of the patch, and the rest are test and documentation changes.
I hope this addresses your concerns.
Best regards,
jitendra
On 10/28/17, 2:00 PM, "Konstantin Shvachko" <sh...@gmail.com> wrote:
Hey guys,
It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.
2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-11118.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.
To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.
I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.
Thanks,
--Konstantin
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Ravi Prakash <ra...@gmail.com>.
Hi folks!
Thank you for sharing the design docs and the tremendous amount of work
that has gone into Ozone. I'm grateful that atleast someone is trying to
drastically improve HDFS.
*If* there is a meeting to discuss this merge, could I please also be
invited?
Have we ever thought about distributing the Namenode metadata across nodes
dynamically based on load and RPC times (unlike static federation that we
have now)?
Also, I think a major feature that HDFS still lacks (and a lot of our users
ask for) is BCP / Disaster Recovery. I only bring this up to see if the
choice of proposed design would have implications for that later on.....
Thanks,
Ravi
On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia <sa...@gmail.com> wrote:
> Konstantine,
> Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
> that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> > On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
> >
> > Hey guys,
> >
> > It is an interesting question whether Ozone should be a part of Hadoop.
> > There are two main reasons why I think it should not.
> >
> > 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> > sizable community behind, it looks to me like a whole new project.
> > It is essentially a new storage system, with different (than HDFS)
> > architecture, separate S3-like APIs. This is really great - the World
> sure
> > needs more distributed file systems. But it is not clear why Ozone should
> > co-exist with HDFS under the same roof.
> >
> > 2. Ozone is probably just the first step in rebuilding HDFS under a new
> > architecture. With the next steps presumably being HDFS-10419 and
> > HDFS-11118.
> > The design doc for the new architecture has never been published. I can
> > only assume based on some presentations and personal communications that
> > the idea is to use Ozone as a block storage, and re-implement NameNode,
> so
> > that it stores only a partial namesapce in memory, while the bulk of it
> > (cold data) is persisted to a local storage.
> > Such architecture makes me wonder if it solves Hadoop's main problems.
> > There are two main limitations in HDFS:
> > a. The throughput of Namespace operations. Which is limited by the
> number
> > of RPCs the NameNode can handle
> > b. The number of objects (files + blocks) the system can maintain. Which
> > is limited by the memory size of the NameNode.
> > The RPC performance (a) is more important for Hadoop scalability than the
> > object count (b). The read RPCs being the main priority.
> > The new architecture targets the object count problem, but in the expense
> > of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> > Also based on the use patterns on our large clusters we read up to 90% of
> > the data we write, so cold data is a small fraction and most of it must
> be
> > cached.
> >
> > To summarize:
> > - Ozone is a big enough system to deserve its own project.
> > - The architecture that Ozone leads to does not seem to solve the
> intrinsic
> > problems of current HDFS.
> >
> > I will post my opinion in the Ozone jira. Should be more convenient to
> > discuss it there for further reference.
> >
> > Thanks,
> > --Konstantin
> >
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >> 1. All services, management scripts
> >> 2. Object store APIs, exposed via both REST and RPC
> >> 3. Master service UIs, command line interfaces
> >> 4. Pluggable pipeline Integration
> >> 5. Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >> 6. Corona - a load generator for Ozone.
> >> 7. Essential documentation added to Hadoop site.
> >> 8. Version specific Ozone Documentation, accessible via service UI.
> >> 9. Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confl
> >> uence/display/HADOOP/Dev+cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ contributors/committers from
> >> different countries and affiliations. We also want to thank the large
> >> number of community members who were supportive of our efforts and
> >> contributed ideas and participated in the design of ozone.
> >>
> >>
> >> Please share your thoughts, thanks!
> >>
> >>
> >> -- Weiwei Yang
> >>
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >> 1. All services, management scripts
> >> 2. Object store APIs, exposed via both REST and RPC
> >> 3. Master service UIs, command line interfaces
> >> 4. Pluggable pipeline Integration
> >> 5. Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >> 6. Corona - a load generator for Ozone.
> >> 7. Essential documentation added to Hadoop site.
> >> 8. Version specific Ozone Documentation, accessible via service UI.
> >> 9. Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> >> cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ contributors/committers from
> >> different countries and affiliations. We also want to thank the large
> >> number of community members who were supportive of our efforts and
> >> contributed ideas and participated in the design of ozone.
> >>
> >>
> >> Please share your thoughts, thanks!
> >>
> >>
> >> -- Weiwei Yang
> >>
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi Sanjay,
Read your doc. I clearly see the value of Ozone with your use cases, but I
agree with Stack and others the question why it should be a part of Hadoop
isn't clear. More details in the jira:
https://issues.apache.org/jira/browse/HDFS-7240?focusedCommentId=16239313&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16239313
Thanks,
--Konstantin
On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia <sa...@gmail.com> wrote:
> Konstantine,
> Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
> that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
>
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Ravi Prakash <ra...@gmail.com>.
Hi folks!
Thank you for sharing the design docs and the tremendous amount of work
that has gone into Ozone. I'm grateful that atleast someone is trying to
drastically improve HDFS.
*If* there is a meeting to discuss this merge, could I please also be
invited?
Have we ever thought about distributing the Namenode metadata across nodes
dynamically based on load and RPC times (unlike static federation that we
have now)?
Also, I think a major feature that HDFS still lacks (and a lot of our users
ask for) is BCP / Disaster Recovery. I only bring this up to see if the
choice of proposed design would have implications for that later on.....
Thanks,
Ravi
On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia <sa...@gmail.com> wrote:
> Konstantine,
> Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
> that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> > On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
> >
> > Hey guys,
> >
> > It is an interesting question whether Ozone should be a part of Hadoop.
> > There are two main reasons why I think it should not.
> >
> > 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> > sizable community behind, it looks to me like a whole new project.
> > It is essentially a new storage system, with different (than HDFS)
> > architecture, separate S3-like APIs. This is really great - the World
> sure
> > needs more distributed file systems. But it is not clear why Ozone should
> > co-exist with HDFS under the same roof.
> >
> > 2. Ozone is probably just the first step in rebuilding HDFS under a new
> > architecture. With the next steps presumably being HDFS-10419 and
> > HDFS-11118.
> > The design doc for the new architecture has never been published. I can
> > only assume based on some presentations and personal communications that
> > the idea is to use Ozone as a block storage, and re-implement NameNode,
> so
> > that it stores only a partial namesapce in memory, while the bulk of it
> > (cold data) is persisted to a local storage.
> > Such architecture makes me wonder if it solves Hadoop's main problems.
> > There are two main limitations in HDFS:
> > a. The throughput of Namespace operations. Which is limited by the
> number
> > of RPCs the NameNode can handle
> > b. The number of objects (files + blocks) the system can maintain. Which
> > is limited by the memory size of the NameNode.
> > The RPC performance (a) is more important for Hadoop scalability than the
> > object count (b). The read RPCs being the main priority.
> > The new architecture targets the object count problem, but in the expense
> > of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> > Also based on the use patterns on our large clusters we read up to 90% of
> > the data we write, so cold data is a small fraction and most of it must
> be
> > cached.
> >
> > To summarize:
> > - Ozone is a big enough system to deserve its own project.
> > - The architecture that Ozone leads to does not seem to solve the
> intrinsic
> > problems of current HDFS.
> >
> > I will post my opinion in the Ozone jira. Should be more convenient to
> > discuss it there for further reference.
> >
> > Thanks,
> > --Konstantin
> >
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >> 1. All services, management scripts
> >> 2. Object store APIs, exposed via both REST and RPC
> >> 3. Master service UIs, command line interfaces
> >> 4. Pluggable pipeline Integration
> >> 5. Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >> 6. Corona - a load generator for Ozone.
> >> 7. Essential documentation added to Hadoop site.
> >> 8. Version specific Ozone Documentation, accessible via service UI.
> >> 9. Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confl
> >> uence/display/HADOOP/Dev+cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ contributors/committers from
> >> different countries and affiliations. We also want to thank the large
> >> number of community members who were supportive of our efforts and
> >> contributed ideas and participated in the design of ozone.
> >>
> >>
> >> Please share your thoughts, thanks!
> >>
> >>
> >> -- Weiwei Yang
> >>
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >> 1. All services, management scripts
> >> 2. Object store APIs, exposed via both REST and RPC
> >> 3. Master service UIs, command line interfaces
> >> 4. Pluggable pipeline Integration
> >> 5. Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >> 6. Corona - a load generator for Ozone.
> >> 7. Essential documentation added to Hadoop site.
> >> 8. Version specific Ozone Documentation, accessible via service UI.
> >> 9. Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> >> cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ contributors/committers from
> >> different countries and affiliations. We also want to thank the large
> >> number of community members who were supportive of our efforts and
> >> contributed ideas and participated in the design of ozone.
> >>
> >>
> >> Please share your thoughts, thanks!
> >>
> >>
> >> -- Weiwei Yang
> >>
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Ravi Prakash <ra...@gmail.com>.
Hi folks!
Thank you for sharing the design docs and the tremendous amount of work
that has gone into Ozone. I'm grateful that atleast someone is trying to
drastically improve HDFS.
*If* there is a meeting to discuss this merge, could I please also be
invited?
Have we ever thought about distributing the Namenode metadata across nodes
dynamically based on load and RPC times (unlike static federation that we
have now)?
Also, I think a major feature that HDFS still lacks (and a lot of our users
ask for) is BCP / Disaster Recovery. I only bring this up to see if the
choice of proposed design would have implications for that later on.....
Thanks,
Ravi
On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia <sa...@gmail.com> wrote:
> Konstantine,
> Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
> that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> > On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
> >
> > Hey guys,
> >
> > It is an interesting question whether Ozone should be a part of Hadoop.
> > There are two main reasons why I think it should not.
> >
> > 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> > sizable community behind, it looks to me like a whole new project.
> > It is essentially a new storage system, with different (than HDFS)
> > architecture, separate S3-like APIs. This is really great - the World
> sure
> > needs more distributed file systems. But it is not clear why Ozone should
> > co-exist with HDFS under the same roof.
> >
> > 2. Ozone is probably just the first step in rebuilding HDFS under a new
> > architecture. With the next steps presumably being HDFS-10419 and
> > HDFS-11118.
> > The design doc for the new architecture has never been published. I can
> > only assume based on some presentations and personal communications that
> > the idea is to use Ozone as a block storage, and re-implement NameNode,
> so
> > that it stores only a partial namesapce in memory, while the bulk of it
> > (cold data) is persisted to a local storage.
> > Such architecture makes me wonder if it solves Hadoop's main problems.
> > There are two main limitations in HDFS:
> > a. The throughput of Namespace operations. Which is limited by the
> number
> > of RPCs the NameNode can handle
> > b. The number of objects (files + blocks) the system can maintain. Which
> > is limited by the memory size of the NameNode.
> > The RPC performance (a) is more important for Hadoop scalability than the
> > object count (b). The read RPCs being the main priority.
> > The new architecture targets the object count problem, but in the expense
> > of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> > Also based on the use patterns on our large clusters we read up to 90% of
> > the data we write, so cold data is a small fraction and most of it must
> be
> > cached.
> >
> > To summarize:
> > - Ozone is a big enough system to deserve its own project.
> > - The architecture that Ozone leads to does not seem to solve the
> intrinsic
> > problems of current HDFS.
> >
> > I will post my opinion in the Ozone jira. Should be more convenient to
> > discuss it there for further reference.
> >
> > Thanks,
> > --Konstantin
> >
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >> 1. All services, management scripts
> >> 2. Object store APIs, exposed via both REST and RPC
> >> 3. Master service UIs, command line interfaces
> >> 4. Pluggable pipeline Integration
> >> 5. Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >> 6. Corona - a load generator for Ozone.
> >> 7. Essential documentation added to Hadoop site.
> >> 8. Version specific Ozone Documentation, accessible via service UI.
> >> 9. Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confl
> >> uence/display/HADOOP/Dev+cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ contributors/committers from
> >> different countries and affiliations. We also want to thank the large
> >> number of community members who were supportive of our efforts and
> >> contributed ideas and participated in the design of ozone.
> >>
> >>
> >> Please share your thoughts, thanks!
> >>
> >>
> >> -- Weiwei Yang
> >>
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >> 1. All services, management scripts
> >> 2. Object store APIs, exposed via both REST and RPC
> >> 3. Master service UIs, command line interfaces
> >> 4. Pluggable pipeline Integration
> >> 5. Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >> 6. Corona - a load generator for Ozone.
> >> 7. Essential documentation added to Hadoop site.
> >> 8. Version specific Ozone Documentation, accessible via service UI.
> >> 9. Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> >> cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ contributors/committers from
> >> different countries and affiliations. We also want to thank the large
> >> number of community members who were supportive of our efforts and
> >> contributed ideas and participated in the design of ozone.
> >>
> >>
> >> Please share your thoughts, thanks!
> >>
> >>
> >> -- Weiwei Yang
> >>
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi Sanjay,
Read your doc. I clearly see the value of Ozone with your use cases, but I
agree with Stack and others the question why it should be a part of Hadoop
isn't clear. More details in the jira:
https://issues.apache.org/jira/browse/HDFS-7240?focusedCommentId=16239313&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16239313
Thanks,
--Konstantin
On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia <sa...@gmail.com> wrote:
> Konstantine,
> Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
> that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
>
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Ravi Prakash <ra...@gmail.com>.
Hi folks!
Thank you for sharing the design docs and the tremendous amount of work
that has gone into Ozone. I'm grateful that atleast someone is trying to
drastically improve HDFS.
*If* there is a meeting to discuss this merge, could I please also be
invited?
Have we ever thought about distributing the Namenode metadata across nodes
dynamically based on load and RPC times (unlike static federation that we
have now)?
Also, I think a major feature that HDFS still lacks (and a lot of our users
ask for) is BCP / Disaster Recovery. I only bring this up to see if the
choice of proposed design would have implications for that later on.....
Thanks,
Ravi
On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia <sa...@gmail.com> wrote:
> Konstantine,
> Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
> that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> > On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
> >
> > Hey guys,
> >
> > It is an interesting question whether Ozone should be a part of Hadoop.
> > There are two main reasons why I think it should not.
> >
> > 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> > sizable community behind, it looks to me like a whole new project.
> > It is essentially a new storage system, with different (than HDFS)
> > architecture, separate S3-like APIs. This is really great - the World
> sure
> > needs more distributed file systems. But it is not clear why Ozone should
> > co-exist with HDFS under the same roof.
> >
> > 2. Ozone is probably just the first step in rebuilding HDFS under a new
> > architecture. With the next steps presumably being HDFS-10419 and
> > HDFS-11118.
> > The design doc for the new architecture has never been published. I can
> > only assume based on some presentations and personal communications that
> > the idea is to use Ozone as a block storage, and re-implement NameNode,
> so
> > that it stores only a partial namesapce in memory, while the bulk of it
> > (cold data) is persisted to a local storage.
> > Such architecture makes me wonder if it solves Hadoop's main problems.
> > There are two main limitations in HDFS:
> > a. The throughput of Namespace operations. Which is limited by the
> number
> > of RPCs the NameNode can handle
> > b. The number of objects (files + blocks) the system can maintain. Which
> > is limited by the memory size of the NameNode.
> > The RPC performance (a) is more important for Hadoop scalability than the
> > object count (b). The read RPCs being the main priority.
> > The new architecture targets the object count problem, but in the expense
> > of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> > Also based on the use patterns on our large clusters we read up to 90% of
> > the data we write, so cold data is a small fraction and most of it must
> be
> > cached.
> >
> > To summarize:
> > - Ozone is a big enough system to deserve its own project.
> > - The architecture that Ozone leads to does not seem to solve the
> intrinsic
> > problems of current HDFS.
> >
> > I will post my opinion in the Ozone jira. Should be more convenient to
> > discuss it there for further reference.
> >
> > Thanks,
> > --Konstantin
> >
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >> 1. All services, management scripts
> >> 2. Object store APIs, exposed via both REST and RPC
> >> 3. Master service UIs, command line interfaces
> >> 4. Pluggable pipeline Integration
> >> 5. Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >> 6. Corona - a load generator for Ozone.
> >> 7. Essential documentation added to Hadoop site.
> >> 8. Version specific Ozone Documentation, accessible via service UI.
> >> 9. Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confl
> >> uence/display/HADOOP/Dev+cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ contributors/committers from
> >> different countries and affiliations. We also want to thank the large
> >> number of community members who were supportive of our efforts and
> >> contributed ideas and participated in the design of ozone.
> >>
> >>
> >> Please share your thoughts, thanks!
> >>
> >>
> >> -- Weiwei Yang
> >>
> >
> >
> > On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
> >
> >> Hello everyone,
> >>
> >>
> >> I would like to start this thread to discuss merging Ozone (HDFS-7240)
> to
> >> trunk. This feature implements an object store which can co-exist with
> >> HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> >> varying from 1 to 100 data nodes.
> >>
> >>
> >>
> >> The merge payload includes the following:
> >>
> >> 1. All services, management scripts
> >> 2. Object store APIs, exposed via both REST and RPC
> >> 3. Master service UIs, command line interfaces
> >> 4. Pluggable pipeline Integration
> >> 5. Ozone File System (Hadoop compatible file system implementation,
> >> passes all FileSystem contract tests)
> >> 6. Corona - a load generator for Ozone.
> >> 7. Essential documentation added to Hadoop site.
> >> 8. Version specific Ozone Documentation, accessible via service UI.
> >> 9. Docker support for ozone, which enables faster development cycles.
> >>
> >>
> >> To build Ozone and run ozone using docker, please follow instructions in
> >> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> >> cluster+with+docker.
> >>
> >>
> >> We have built a passionate and diverse community to drive this feature
> >> development. As a team, we have achieved significant progress in past 3
> >> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> >> have resolved almost 400 JIRAs by 20+ contributors/committers from
> >> different countries and affiliations. We also want to thank the large
> >> number of community members who were supportive of our efforts and
> >> contributed ideas and participated in the design of ozone.
> >>
> >>
> >> Please share your thoughts, thanks!
> >>
> >>
> >> -- Weiwei Yang
> >>
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi Sanjay,
Read your doc. I clearly see the value of Ozone with your use cases, but I
agree with Stack and others the question why it should be a part of Hadoop
isn't clear. More details in the jira:
https://issues.apache.org/jira/browse/HDFS-7240?focusedCommentId=16239313&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16239313
Thanks,
--Konstantin
On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia <sa...@gmail.com> wrote:
> Konstantine,
> Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
> that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
>
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hi Sanjay,
Read your doc. I clearly see the value of Ozone with your use cases, but I
agree with Stack and others the question why it should be a part of Hadoop
isn't clear. More details in the jira:
https://issues.apache.org/jira/browse/HDFS-7240?focusedCommentId=16239313&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16239313
Thanks,
--Konstantin
On Fri, Nov 3, 2017 at 1:56 PM, sanjay Radia <sa...@gmail.com> wrote:
> Konstantine,
> Thanks for your comments, questions and feedback. I have attached a
> document to the HDFS-7240 jira
> that explains a design for scaling HDFS and how Ozone paves the way
> towards the full solution.
>
>
> https://issues.apache.org/jira/secure/attachment/
> 12895963/HDFS%20Scalability%20and%20Ozone.pdf
>
>
> sanjay
>
>
>
>
> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
>
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
>
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Konstantine,
Thanks for your comments, questions and feedback. I have attached a document to the HDFS-7240 jira
that explains a design for scaling HDFS and how Ozone paves the way towards the full solution.
https://issues.apache.org/jira/secure/attachment/12895963/HDFS%20Scalability%20and%20Ozone.pdf
sanjay
> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com> wrote:
>
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
>
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
>
>> Hello everyone,
>>
>>
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>>
>>
>>
>> The merge payload includes the following:
>>
>> 1. All services, management scripts
>> 2. Object store APIs, exposed via both REST and RPC
>> 3. Master service UIs, command line interfaces
>> 4. Pluggable pipeline Integration
>> 5. Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>> 6. Corona - a load generator for Ozone.
>> 7. Essential documentation added to Hadoop site.
>> 8. Version specific Ozone Documentation, accessible via service UI.
>> 9. Docker support for ozone, which enables faster development cycles.
>>
>>
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confl
>> uence/display/HADOOP/Dev+cluster+with+docker.
>>
>>
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>>
>>
>> Please share your thoughts, thanks!
>>
>>
>> -- Weiwei Yang
>>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
>
>> Hello everyone,
>>
>>
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>>
>>
>>
>> The merge payload includes the following:
>>
>> 1. All services, management scripts
>> 2. Object store APIs, exposed via both REST and RPC
>> 3. Master service UIs, command line interfaces
>> 4. Pluggable pipeline Integration
>> 5. Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>> 6. Corona - a load generator for Ozone.
>> 7. Essential documentation added to Hadoop site.
>> 8. Version specific Ozone Documentation, accessible via service UI.
>> 9. Docker support for ozone, which enables faster development cycles.
>>
>>
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
>> cluster+with+docker.
>>
>>
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>>
>>
>> Please share your thoughts, thanks!
>>
>>
>> -- Weiwei Yang
>>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Konstantine,
Thanks for your comments, questions and feedback. I have attached a document to the HDFS-7240 jira
that explains a design for scaling HDFS and how Ozone paves the way towards the full solution.
https://issues.apache.org/jira/secure/attachment/12895963/HDFS%20Scalability%20and%20Ozone.pdf
sanjay
> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com> wrote:
>
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
>
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
>
>> Hello everyone,
>>
>>
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>>
>>
>>
>> The merge payload includes the following:
>>
>> 1. All services, management scripts
>> 2. Object store APIs, exposed via both REST and RPC
>> 3. Master service UIs, command line interfaces
>> 4. Pluggable pipeline Integration
>> 5. Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>> 6. Corona - a load generator for Ozone.
>> 7. Essential documentation added to Hadoop site.
>> 8. Version specific Ozone Documentation, accessible via service UI.
>> 9. Docker support for ozone, which enables faster development cycles.
>>
>>
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confl
>> uence/display/HADOOP/Dev+cluster+with+docker.
>>
>>
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>>
>>
>> Please share your thoughts, thanks!
>>
>>
>> -- Weiwei Yang
>>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
>
>> Hello everyone,
>>
>>
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>>
>>
>>
>> The merge payload includes the following:
>>
>> 1. All services, management scripts
>> 2. Object store APIs, exposed via both REST and RPC
>> 3. Master service UIs, command line interfaces
>> 4. Pluggable pipeline Integration
>> 5. Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>> 6. Corona - a load generator for Ozone.
>> 7. Essential documentation added to Hadoop site.
>> 8. Version specific Ozone Documentation, accessible via service UI.
>> 9. Docker support for ozone, which enables faster development cycles.
>>
>>
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
>> cluster+with+docker.
>>
>>
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>>
>>
>> Please share your thoughts, thanks!
>>
>>
>> -- Weiwei Yang
>>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
> At a minimum, it should at least be using it’s own maven module for a lot of the bits that generates it’s own maven jars so that we can split this functionality up at build/test time.
I expected this to be the case, but looks like it isn't.
There's lot of value in splitting the HDFS code into smaller modules. Definitely newer code like Ozone.
When we did this for YARN, initially there were concerns about module proliferation, but looking back, my observations have been that it has done us far more good than expected. Starting with the fact that we had clients and servers modularized independently, as well as servers from other servers, with far cleaner contracts than what we had in Hadoop 1 world.
Thanks
+Vinod
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
> At a minimum, it should at least be using it’s own maven module for a lot of the bits that generates it’s own maven jars so that we can split this functionality up at build/test time.
I expected this to be the case, but looks like it isn't.
There's lot of value in splitting the HDFS code into smaller modules. Definitely newer code like Ozone.
When we did this for YARN, initially there were concerns about module proliferation, but looking back, my observations have been that it has done us far more good than expected. Starting with the fact that we had clients and servers modularized independently, as well as servers from other servers, with far cleaner contracts than what we had in Hadoop 1 world.
Thanks
+Vinod
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
> At a minimum, it should at least be using it’s own maven module for a lot of the bits that generates it’s own maven jars so that we can split this functionality up at build/test time.
I expected this to be the case, but looks like it isn't.
There's lot of value in splitting the HDFS code into smaller modules. Definitely newer code like Ozone.
When we did this for YARN, initially there were concerns about module proliferation, but looking back, my observations have been that it has done us far more good than expected. Starting with the fact that we had clients and servers modularized independently, as well as servers from other servers, with far cleaner contracts than what we had in Hadoop 1 world.
Thanks
+Vinod
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
> At a minimum, it should at least be using it’s own maven module for a lot of the bits that generates it’s own maven jars so that we can split this functionality up at build/test time.
I expected this to be the case, but looks like it isn't.
There's lot of value in splitting the HDFS code into smaller modules. Definitely newer code like Ozone.
When we did this for YARN, initially there were concerns about module proliferation, but looking back, my observations have been that it has done us far more good than expected. Starting with the fact that we had clients and servers modularized independently, as well as servers from other servers, with far cleaner contracts than what we had in Hadoop 1 world.
Thanks
+Vinod
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Nov 3, 2017, at 12:08 PM, Stack <st...@duboce.net> wrote:
>
> On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> It is an interesting question whether Ozone should be a part of Hadoop.
>
> I don't see a direct answer to this question. Is there one? Pardon me if
> I've not seen it but I'm interested in the response.
+1
Given:
* a completely different set of config files (ozone-site.xml, etc)
* package name is org.apache.hadoop.ozone, not org.apache.hadoop.hdfs.ozone
… it doesn’t really seem to want to be part of HDFS, much less Hadoop.
Plus hadoop-hdfs-project/hadoop-hdfs is already a battle zone when it comes to unit tests, dependencies, etc [*]
At a minimum, it should at least be using it’s own maven module for a lot of the bits that generates it’s own maven jars so that we can split this functionality up at build/test time.
At a higher level, this feels a lot like the design decisions that were made around yarn-native-services. This feature is either part of HDFS or it’s not. Pick one. Doing both is incredibly confusing for everyone outside of the branch.
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Nov 3, 2017, at 12:08 PM, Stack <st...@duboce.net> wrote:
>
> On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> It is an interesting question whether Ozone should be a part of Hadoop.
>
> I don't see a direct answer to this question. Is there one? Pardon me if
> I've not seen it but I'm interested in the response.
+1
Given:
* a completely different set of config files (ozone-site.xml, etc)
* package name is org.apache.hadoop.ozone, not org.apache.hadoop.hdfs.ozone
… it doesn’t really seem to want to be part of HDFS, much less Hadoop.
Plus hadoop-hdfs-project/hadoop-hdfs is already a battle zone when it comes to unit tests, dependencies, etc [*]
At a minimum, it should at least be using it’s own maven module for a lot of the bits that generates it’s own maven jars so that we can split this functionality up at build/test time.
At a higher level, this feels a lot like the design decisions that were made around yarn-native-services. This feature is either part of HDFS or it’s not. Pick one. Doing both is incredibly confusing for everyone outside of the branch.
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Nov 3, 2017, at 12:08 PM, Stack <st...@duboce.net> wrote:
>
> On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> It is an interesting question whether Ozone should be a part of Hadoop.
>
> I don't see a direct answer to this question. Is there one? Pardon me if
> I've not seen it but I'm interested in the response.
+1
Given:
* a completely different set of config files (ozone-site.xml, etc)
* package name is org.apache.hadoop.ozone, not org.apache.hadoop.hdfs.ozone
… it doesn’t really seem to want to be part of HDFS, much less Hadoop.
Plus hadoop-hdfs-project/hadoop-hdfs is already a battle zone when it comes to unit tests, dependencies, etc [*]
At a minimum, it should at least be using it’s own maven module for a lot of the bits that generates it’s own maven jars so that we can split this functionality up at build/test time.
At a higher level, this feels a lot like the design decisions that were made around yarn-native-services. This feature is either part of HDFS or it’s not. Pick one. Doing both is incredibly confusing for everyone outside of the branch.
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Allen Wittenauer <aw...@effectivemachines.com>.
> On Nov 3, 2017, at 12:08 PM, Stack <st...@duboce.net> wrote:
>
> On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
> wrote:
>
>> It is an interesting question whether Ozone should be a part of Hadoop.
>
> I don't see a direct answer to this question. Is there one? Pardon me if
> I've not seen it but I'm interested in the response.
+1
Given:
* a completely different set of config files (ozone-site.xml, etc)
* package name is org.apache.hadoop.ozone, not org.apache.hadoop.hdfs.ozone
… it doesn’t really seem to want to be part of HDFS, much less Hadoop.
Plus hadoop-hdfs-project/hadoop-hdfs is already a battle zone when it comes to unit tests, dependencies, etc [*]
At a minimum, it should at least be using it’s own maven module for a lot of the bits that generates it’s own maven jars so that we can split this functionality up at build/test time.
At a higher level, this feels a lot like the design decisions that were made around yarn-native-services. This feature is either part of HDFS or it’s not. Pick one. Doing both is incredibly confusing for everyone outside of the branch.
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Stack <st...@duboce.net>.
On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
>
I don't see a direct answer to this question. Is there one? Pardon me if
I've not seen it but I'm interested in the response.
I ask because IMO the "Hadoop" project is over-stuffed already. Just see
the length of the cc list on this email. Ozone could be standalone. It is a
coherent enough effort.
Thanks,
St.Ack
> There are two main reasons why I think it should not.
>
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> > 1. All services, management scripts
> > 2. Object store APIs, exposed via both REST and RPC
> > 3. Master service UIs, command line interfaces
> > 4. Pluggable pipeline Integration
> > 5. Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> > 6. Corona - a load generator for Ozone.
> > 7. Essential documentation added to Hadoop site.
> > 8. Version specific Ozone Documentation, accessible via service UI.
> > 9. Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confl
> > uence/display/HADOOP/Dev+cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> > 1. All services, management scripts
> > 2. Object store APIs, exposed via both REST and RPC
> > 3. Master service UIs, command line interfaces
> > 4. Pluggable pipeline Integration
> > 5. Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> > 6. Corona - a load generator for Ozone.
> > 7. Essential documentation added to Hadoop site.
> > 8. Version specific Ozone Documentation, accessible via service UI.
> > 9. Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> > cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Jitendra Pandey <ji...@hortonworks.com>.
Hi Konstantin,
Thank you for taking out time to review ozone. I appreciate your comments and questions.
> There are two main limitations in HDFS
> a) The throughput of Namespace operations. Which is limited by the
> number of RPCs the NameNode can handle
> b) The number of objects (files + blocks) the system can maintain.
> Which is limited by the memory size of the NameNode.
I agree completely. We believe ozone attempts to address both these issues for HDFS.
Let us look at the Number of objects problem. Ozone directly addresses the scalability of number of blocks by introducing storage containers that can hold multiple blocks together. The earlier efforts on this were complicated by the fact that block manager and namespace are intertwined in HDFS Namenode. There have been efforts in past to separate block manager from namespace for e.g. HDFS-5477. Ozone addresses this problem by cleanly separating the block layer. Separation of block layer also addresses the file/directories scalability because it frees up the blockmap from the namenode.
Separate block layer relieves namenode from handling block reports, IBRs, heartbeats, replication monitor etc, and thus reduces the contention on FSNamesystem lock and significantly reduces the GC pressure on the namenode. These improvements will greatly help the RPC performance of the Namenode.
> Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118. The design doc for the new architecture has never been
> published.
We do believe that Namenode can leverage the ozone’s storage container layer, however, that is also a big effort. We would like to first have block layer stabilized in ozone before taking that up. However, we would certainly support any community effort on that, and in fact it was brought up in last BoF session at the summit.
Big data is evolving rapidly. We see our customers needing scalable file systems, Objects stores(like S3) and Block Store(for docker and VMs). Ozone improves HDFS in two ways. It addresses throughput and scale issues of HDFS, and enriches it with newer capabilities.
> Ozone is a big enough system to deserve its own project.
I took a quick look at the core code in ozone and the cloc command reports 22,511 lines of functionality changes in Java.
This patch also brings in web framework code like Angular.js and that brings in bunch of css and js files that contribute to the size of the patch, and the rest are test and documentation changes.
I hope this addresses your concerns.
Best regards,
jitendra
On 10/28/17, 2:00 PM, "Konstantin Shvachko" <sh...@gmail.com> wrote:
Hey guys,
It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.
2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-11118.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.
To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.
I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.
Thanks,
--Konstantin
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Jitendra Pandey <ji...@hortonworks.com>.
Hi Konstantin,
Thank you for taking out time to review ozone. I appreciate your comments and questions.
> There are two main limitations in HDFS
> a) The throughput of Namespace operations. Which is limited by the
> number of RPCs the NameNode can handle
> b) The number of objects (files + blocks) the system can maintain.
> Which is limited by the memory size of the NameNode.
I agree completely. We believe ozone attempts to address both these issues for HDFS.
Let us look at the Number of objects problem. Ozone directly addresses the scalability of number of blocks by introducing storage containers that can hold multiple blocks together. The earlier efforts on this were complicated by the fact that block manager and namespace are intertwined in HDFS Namenode. There have been efforts in past to separate block manager from namespace for e.g. HDFS-5477. Ozone addresses this problem by cleanly separating the block layer. Separation of block layer also addresses the file/directories scalability because it frees up the blockmap from the namenode.
Separate block layer relieves namenode from handling block reports, IBRs, heartbeats, replication monitor etc, and thus reduces the contention on FSNamesystem lock and significantly reduces the GC pressure on the namenode. These improvements will greatly help the RPC performance of the Namenode.
> Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118. The design doc for the new architecture has never been
> published.
We do believe that Namenode can leverage the ozone’s storage container layer, however, that is also a big effort. We would like to first have block layer stabilized in ozone before taking that up. However, we would certainly support any community effort on that, and in fact it was brought up in last BoF session at the summit.
Big data is evolving rapidly. We see our customers needing scalable file systems, Objects stores(like S3) and Block Store(for docker and VMs). Ozone improves HDFS in two ways. It addresses throughput and scale issues of HDFS, and enriches it with newer capabilities.
> Ozone is a big enough system to deserve its own project.
I took a quick look at the core code in ozone and the cloc command reports 22,511 lines of functionality changes in Java.
This patch also brings in web framework code like Angular.js and that brings in bunch of css and js files that contribute to the size of the patch, and the rest are test and documentation changes.
I hope this addresses your concerns.
Best regards,
jitendra
On 10/28/17, 2:00 PM, "Konstantin Shvachko" <sh...@gmail.com> wrote:
Hey guys,
It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.
2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-11118.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.
To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.
I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.
Thanks,
--Konstantin
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Konstantine,
Thanks for your comments, questions and feedback. I have attached a document to the HDFS-7240 jira
that explains a design for scaling HDFS and how Ozone paves the way towards the full solution.
https://issues.apache.org/jira/secure/attachment/12895963/HDFS%20Scalability%20and%20Ozone.pdf
sanjay
> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com> wrote:
>
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
>
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
>
>> Hello everyone,
>>
>>
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>>
>>
>>
>> The merge payload includes the following:
>>
>> 1. All services, management scripts
>> 2. Object store APIs, exposed via both REST and RPC
>> 3. Master service UIs, command line interfaces
>> 4. Pluggable pipeline Integration
>> 5. Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>> 6. Corona - a load generator for Ozone.
>> 7. Essential documentation added to Hadoop site.
>> 8. Version specific Ozone Documentation, accessible via service UI.
>> 9. Docker support for ozone, which enables faster development cycles.
>>
>>
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confl
>> uence/display/HADOOP/Dev+cluster+with+docker.
>>
>>
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>>
>>
>> Please share your thoughts, thanks!
>>
>>
>> -- Weiwei Yang
>>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
>
>> Hello everyone,
>>
>>
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>>
>>
>>
>> The merge payload includes the following:
>>
>> 1. All services, management scripts
>> 2. Object store APIs, exposed via both REST and RPC
>> 3. Master service UIs, command line interfaces
>> 4. Pluggable pipeline Integration
>> 5. Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>> 6. Corona - a load generator for Ozone.
>> 7. Essential documentation added to Hadoop site.
>> 8. Version specific Ozone Documentation, accessible via service UI.
>> 9. Docker support for ozone, which enables faster development cycles.
>>
>>
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
>> cluster+with+docker.
>>
>>
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>>
>>
>> Please share your thoughts, thanks!
>>
>>
>> -- Weiwei Yang
>>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Konstantine,
Thanks for your comments, questions and feedback. I have attached a document to the HDFS-7240 jira
that explains a design for scaling HDFS and how Ozone paves the way towards the full solution.
https://issues.apache.org/jira/secure/attachment/12895963/HDFS%20Scalability%20and%20Ozone.pdf
sanjay
> On Oct 28, 2017, at 2:00 PM, Konstantin Shvachko <sh...@gmail.com> wrote:
>
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
> There are two main reasons why I think it should not.
>
> 1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
>
>> Hello everyone,
>>
>>
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>>
>>
>>
>> The merge payload includes the following:
>>
>> 1. All services, management scripts
>> 2. Object store APIs, exposed via both REST and RPC
>> 3. Master service UIs, command line interfaces
>> 4. Pluggable pipeline Integration
>> 5. Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>> 6. Corona - a load generator for Ozone.
>> 7. Essential documentation added to Hadoop site.
>> 8. Version specific Ozone Documentation, accessible via service UI.
>> 9. Docker support for ozone, which enables faster development cycles.
>>
>>
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confl
>> uence/display/HADOOP/Dev+cluster+with+docker.
>>
>>
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>>
>>
>> Please share your thoughts, thanks!
>>
>>
>> -- Weiwei Yang
>>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
>
>> Hello everyone,
>>
>>
>> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
>> trunk. This feature implements an object store which can co-exist with
>> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
>> varying from 1 to 100 data nodes.
>>
>>
>>
>> The merge payload includes the following:
>>
>> 1. All services, management scripts
>> 2. Object store APIs, exposed via both REST and RPC
>> 3. Master service UIs, command line interfaces
>> 4. Pluggable pipeline Integration
>> 5. Ozone File System (Hadoop compatible file system implementation,
>> passes all FileSystem contract tests)
>> 6. Corona - a load generator for Ozone.
>> 7. Essential documentation added to Hadoop site.
>> 8. Version specific Ozone Documentation, accessible via service UI.
>> 9. Docker support for ozone, which enables faster development cycles.
>>
>>
>> To build Ozone and run ozone using docker, please follow instructions in
>> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
>> cluster+with+docker.
>>
>>
>> We have built a passionate and diverse community to drive this feature
>> development. As a team, we have achieved significant progress in past 3
>> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
>> have resolved almost 400 JIRAs by 20+ contributors/committers from
>> different countries and affiliations. We also want to thank the large
>> number of community members who were supportive of our efforts and
>> contributed ideas and participated in the design of ozone.
>>
>>
>> Please share your thoughts, thanks!
>>
>>
>> -- Weiwei Yang
>>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Stack <st...@duboce.net>.
On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
>
I don't see a direct answer to this question. Is there one? Pardon me if
I've not seen it but I'm interested in the response.
I ask because IMO the "Hadoop" project is over-stuffed already. Just see
the length of the cc list on this email. Ozone could be standalone. It is a
coherent enough effort.
Thanks,
St.Ack
> There are two main reasons why I think it should not.
>
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> > 1. All services, management scripts
> > 2. Object store APIs, exposed via both REST and RPC
> > 3. Master service UIs, command line interfaces
> > 4. Pluggable pipeline Integration
> > 5. Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> > 6. Corona - a load generator for Ozone.
> > 7. Essential documentation added to Hadoop site.
> > 8. Version specific Ozone Documentation, accessible via service UI.
> > 9. Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confl
> > uence/display/HADOOP/Dev+cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> > 1. All services, management scripts
> > 2. Object store APIs, exposed via both REST and RPC
> > 3. Master service UIs, command line interfaces
> > 4. Pluggable pipeline Integration
> > 5. Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> > 6. Corona - a load generator for Ozone.
> > 7. Essential documentation added to Hadoop site.
> > 8. Version specific Ozone Documentation, accessible via service UI.
> > 9. Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> > cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Jitendra Pandey <ji...@hortonworks.com>.
Hi Konstantin,
Thank you for taking out time to review ozone. I appreciate your comments and questions.
> There are two main limitations in HDFS
> a) The throughput of Namespace operations. Which is limited by the
> number of RPCs the NameNode can handle
> b) The number of objects (files + blocks) the system can maintain.
> Which is limited by the memory size of the NameNode.
I agree completely. We believe ozone attempts to address both these issues for HDFS.
Let us look at the Number of objects problem. Ozone directly addresses the scalability of number of blocks by introducing storage containers that can hold multiple blocks together. The earlier efforts on this were complicated by the fact that block manager and namespace are intertwined in HDFS Namenode. There have been efforts in past to separate block manager from namespace for e.g. HDFS-5477. Ozone addresses this problem by cleanly separating the block layer. Separation of block layer also addresses the file/directories scalability because it frees up the blockmap from the namenode.
Separate block layer relieves namenode from handling block reports, IBRs, heartbeats, replication monitor etc, and thus reduces the contention on FSNamesystem lock and significantly reduces the GC pressure on the namenode. These improvements will greatly help the RPC performance of the Namenode.
> Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118. The design doc for the new architecture has never been
> published.
We do believe that Namenode can leverage the ozone’s storage container layer, however, that is also a big effort. We would like to first have block layer stabilized in ozone before taking that up. However, we would certainly support any community effort on that, and in fact it was brought up in last BoF session at the summit.
Big data is evolving rapidly. We see our customers needing scalable file systems, Objects stores(like S3) and Block Store(for docker and VMs). Ozone improves HDFS in two ways. It addresses throughput and scale issues of HDFS, and enriches it with newer capabilities.
> Ozone is a big enough system to deserve its own project.
I took a quick look at the core code in ozone and the cloc command reports 22,511 lines of functionality changes in Java.
This patch also brings in web framework code like Angular.js and that brings in bunch of css and js files that contribute to the size of the patch, and the rest are test and documentation changes.
I hope this addresses your concerns.
Best regards,
jitendra
On 10/28/17, 2:00 PM, "Konstantin Shvachko" <sh...@gmail.com> wrote:
Hey guys,
It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.
2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-11118.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.
To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.
I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.
Thanks,
--Konstantin
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Stack <st...@duboce.net>.
On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
>
I don't see a direct answer to this question. Is there one? Pardon me if
I've not seen it but I'm interested in the response.
I ask because IMO the "Hadoop" project is over-stuffed already. Just see
the length of the cc list on this email. Ozone could be standalone. It is a
coherent enough effort.
Thanks,
St.Ack
> There are two main reasons why I think it should not.
>
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> > 1. All services, management scripts
> > 2. Object store APIs, exposed via both REST and RPC
> > 3. Master service UIs, command line interfaces
> > 4. Pluggable pipeline Integration
> > 5. Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> > 6. Corona - a load generator for Ozone.
> > 7. Essential documentation added to Hadoop site.
> > 8. Version specific Ozone Documentation, accessible via service UI.
> > 9. Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confl
> > uence/display/HADOOP/Dev+cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> > 1. All services, management scripts
> > 2. Object store APIs, exposed via both REST and RPC
> > 3. Master service UIs, command line interfaces
> > 4. Pluggable pipeline Integration
> > 5. Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> > 6. Corona - a load generator for Ozone.
> > 7. Essential documentation added to Hadoop site.
> > 8. Version specific Ozone Documentation, accessible via service UI.
> > 9. Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> > cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Stack <st...@duboce.net>.
On Sat, Oct 28, 2017 at 2:00 PM, Konstantin Shvachko <sh...@gmail.com>
wrote:
> Hey guys,
>
> It is an interesting question whether Ozone should be a part of Hadoop.
>
I don't see a direct answer to this question. Is there one? Pardon me if
I've not seen it but I'm interested in the response.
I ask because IMO the "Hadoop" project is over-stuffed already. Just see
the length of the cc list on this email. Ozone could be standalone. It is a
coherent enough effort.
Thanks,
St.Ack
> There are two main reasons why I think it should not.
>
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
> sizable community behind, it looks to me like a whole new project.
> It is essentially a new storage system, with different (than HDFS)
> architecture, separate S3-like APIs. This is really great - the World sure
> needs more distributed file systems. But it is not clear why Ozone should
> co-exist with HDFS under the same roof.
>
> 2. Ozone is probably just the first step in rebuilding HDFS under a new
> architecture. With the next steps presumably being HDFS-10419 and
> HDFS-11118.
> The design doc for the new architecture has never been published. I can
> only assume based on some presentations and personal communications that
> the idea is to use Ozone as a block storage, and re-implement NameNode, so
> that it stores only a partial namesapce in memory, while the bulk of it
> (cold data) is persisted to a local storage.
> Such architecture makes me wonder if it solves Hadoop's main problems.
> There are two main limitations in HDFS:
> a. The throughput of Namespace operations. Which is limited by the number
> of RPCs the NameNode can handle
> b. The number of objects (files + blocks) the system can maintain. Which
> is limited by the memory size of the NameNode.
> The RPC performance (a) is more important for Hadoop scalability than the
> object count (b). The read RPCs being the main priority.
> The new architecture targets the object count problem, but in the expense
> of the RPC throughput. Which seems to be a wrong resolution of the
> tradeoff.
> Also based on the use patterns on our large clusters we read up to 90% of
> the data we write, so cold data is a small fraction and most of it must be
> cached.
>
> To summarize:
> - Ozone is a big enough system to deserve its own project.
> - The architecture that Ozone leads to does not seem to solve the intrinsic
> problems of current HDFS.
>
> I will post my opinion in the Ozone jira. Should be more convenient to
> discuss it there for further reference.
>
> Thanks,
> --Konstantin
>
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> > 1. All services, management scripts
> > 2. Object store APIs, exposed via both REST and RPC
> > 3. Master service UIs, command line interfaces
> > 4. Pluggable pipeline Integration
> > 5. Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> > 6. Corona - a load generator for Ozone.
> > 7. Essential documentation added to Hadoop site.
> > 8. Version specific Ozone Documentation, accessible via service UI.
> > 9. Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confl
> > uence/display/HADOOP/Dev+cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
>
> On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com>
> wrote:
>
> > Hello everyone,
> >
> >
> > I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> > trunk. This feature implements an object store which can co-exist with
> > HDFS. Ozone is disabled by default. We have tested Ozone with cluster
> sizes
> > varying from 1 to 100 data nodes.
> >
> >
> >
> > The merge payload includes the following:
> >
> > 1. All services, management scripts
> > 2. Object store APIs, exposed via both REST and RPC
> > 3. Master service UIs, command line interfaces
> > 4. Pluggable pipeline Integration
> > 5. Ozone File System (Hadoop compatible file system implementation,
> > passes all FileSystem contract tests)
> > 6. Corona - a load generator for Ozone.
> > 7. Essential documentation added to Hadoop site.
> > 8. Version specific Ozone Documentation, accessible via service UI.
> > 9. Docker support for ozone, which enables faster development cycles.
> >
> >
> > To build Ozone and run ozone using docker, please follow instructions in
> > this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> > cluster+with+docker.
> >
> >
> > We have built a passionate and diverse community to drive this feature
> > development. As a team, we have achieved significant progress in past 3
> > years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> > have resolved almost 400 JIRAs by 20+ contributors/committers from
> > different countries and affiliations. We also want to thank the large
> > number of community members who were supportive of our efforts and
> > contributed ideas and participated in the design of ozone.
> >
> >
> > Please share your thoughts, thanks!
> >
> >
> > -- Weiwei Yang
> >
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hey guys,
It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.
2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-11118.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.
To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.
I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.
Thanks,
--Konstantin
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hey guys,
It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.
2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-11118.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.
To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.
I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.
Thanks,
--Konstantin
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Dear Hadoop Community Members,
We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments. The key questions raised were following
How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
We were asked to provide a security design
There were questions around stability given ozone brings in a large body of code.
Why can’t they be separate projects forever or merged in when production ready?
We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns. Please see the summary below:
The new code base benefits to HDFS scaling and a roadmap has been provided. Summary:
New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful.
We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
Sharing DN daemon to avoid additional operational daemon lifecycle management
Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
Security design, including a threat model and and the solution has been posted.
Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
Merge the new code back into HDFS later will be harder.
The code and goals will diverge further.
We will be taking on extra work to split and then take extra work to merge.
The issues raised today will be raised all the same then.
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by sanjay Radia <sa...@gmail.com>.
Dear Hadoop Community Members,
We had multiple community discussions, a few meetings in smaller groups and also jira discussions with respect to this thread. We express our gratitude for participation and valuable comments. The key questions raised were following
How the new block storage layer and OzoneFS benefit HDFS and we were asked to chalk out a roadmap towards the goal of a scalable namenode working with the new storage layer
We were asked to provide a security design
There were questions around stability given ozone brings in a large body of code.
Why can’t they be separate projects forever or merged in when production ready?
We have responded to all the above questions with detailed explanations and answers on the jira as well as in the discussions. We believe that should sufficiently address community’s concerns. Please see the summary below:
The new code base benefits to HDFS scaling and a roadmap has been provided. Summary:
New block storage layer addresses the scalability of the block layer. We have shown how existing NN can be connected to the new block layer and its benefits. We have shown 2 milestones, 1st milestone is much simpler than 2nd milestone while giving almost the same scaling benefits. Originally we had proposed simply milestone 2 and the community felt that removing the FSN/BM lock was was a fair amount of work and a simpler solution would be useful.
We provide a new K-V namespace called Ozone FS with FileSystem/FileContext plugins to allow the users to use the new system. BTW Hive and Spark work very well on KV-namespaces on the cloud. This will facilitate stabilizing the new block layer.
The new block layer has a new netty based protocol engine in the Datanode which, when stabilized, can be used by the old hdfs block layer. See details below on sharing of code.
Stability impact on the existing HDFS code base and code separation. The new block layer and the OzoneFS are in modules that are separate from old HDFS code - currently there are no calls from HDFS into Ozone except for DN starting the new block layer module if configured to do so. It does not add instability (the instability argument has been raised many times). Over time as we share code, we will ensure that the old HDFS continues to remains stable. (for example we plan to stabilize the new netty based protocol engine in the new block layer before sharing it with HDFS’s old block layer)
In the short term and medium term, the new system and HDFS will be used side-by-side by users. Side by-side usage in the short term for testing and side-by-side in the medium term for actual production use till the new system has feature parity with old HDFS. During this time, sharing the DN daemon and admin functions between the two systems is operationally important:
Sharing DN daemon to avoid additional operational daemon lifecycle management
Common decommissioning of the daemon and DN: One place to decommission for a node and its storage.
Replacing failed disks and internal balancing capacity across disks - this needs to be done for both the current HDFS blocks and the new block-layer blocks.
Balancer: we would like use the same balancer and provide a common way to balance and common management of the bandwidth used for balancing
Security configuration setup - reuse existing set up for DNs rather then a new one for an independent cluster.
Need to easily share the block layer code between the two systems when used side-by-side. Areas where sharing code is desired over time:
Sharing new block layer’s new netty based protocol engine for old HDFS DNs (a long time sore issue for HDFS block layer).
Shallow data copy from old system to new system is practical only if within same project and daemon otherwise have to deal with security setting and coordinations across daemons. Shallow copy is useful as customer migrate from old to new.
Shared disk scheduling in the future and in the short term have a single round robin rather than independent round robins.
While sharing code across projects is technically possible (anything is possible in software), it is significantly harder typically requiring cleaner public apis etc. Sharing within a project though internal APIs is often simpler (such as the protocol engine that we want to share).
Security design, including a threat model and and the solution has been posted.
Temporary Separation and merge later: Several of the comments in the jira have argued that we temporarily separate the two code bases for now and then later merge them when the new code is stable:
If there is agreement to merge later, why bother separating now - there needs to be to be good reasons to separate now. We have addressed the stability and separation of the new code from existing above.
Merge the new code back into HDFS later will be harder.
The code and goals will diverge further.
We will be taking on extra work to split and then take extra work to merge.
The issues raised today will be raised all the same then.
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Steve Loughran <st...@hortonworks.com>.
Wow, big piece of work
1. Where is a PR/branch on github with rendered docs for us to look at?
2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.
thanks
> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hey guys,
It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.
2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-11118.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.
To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.
I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.
Thanks,
--Konstantin
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Steve Loughran <st...@hortonworks.com>.
Wow, big piece of work
1. Where is a PR/branch on github with rendered docs for us to look at?
2. Have you made any public APi changes related to object stores? That's probably something I'll have opinions on more than implementation details.
thanks
> On 19 Oct 2017, at 02:54, Yang Weiwei <ch...@hotmail.com> wrote:
>
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to trunk. This feature implements an object store which can co-exist with HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation, passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature development. As a team, we have achieved significant progress in past 3 years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we have resolved almost 400 JIRAs by 20+ contributors/committers from different countries and affiliations. We also want to thank the large number of community members who were supportive of our efforts and contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [DISCUSSION] Merging HDFS-7240 Object Store (Ozone) to trunk
Posted by Konstantin Shvachko <sh...@gmail.com>.
Hey guys,
It is an interesting question whether Ozone should be a part of Hadoop.
There are two main reasons why I think it should not.
1. With close to 500 sub-tasks, with 6 MB of code changes, and with a
sizable community behind, it looks to me like a whole new project.
It is essentially a new storage system, with different (than HDFS)
architecture, separate S3-like APIs. This is really great - the World sure
needs more distributed file systems. But it is not clear why Ozone should
co-exist with HDFS under the same roof.
2. Ozone is probably just the first step in rebuilding HDFS under a new
architecture. With the next steps presumably being HDFS-10419 and
HDFS-11118.
The design doc for the new architecture has never been published. I can
only assume based on some presentations and personal communications that
the idea is to use Ozone as a block storage, and re-implement NameNode, so
that it stores only a partial namesapce in memory, while the bulk of it
(cold data) is persisted to a local storage.
Such architecture makes me wonder if it solves Hadoop's main problems.
There are two main limitations in HDFS:
a. The throughput of Namespace operations. Which is limited by the number
of RPCs the NameNode can handle
b. The number of objects (files + blocks) the system can maintain. Which
is limited by the memory size of the NameNode.
The RPC performance (a) is more important for Hadoop scalability than the
object count (b). The read RPCs being the main priority.
The new architecture targets the object count problem, but in the expense
of the RPC throughput. Which seems to be a wrong resolution of the tradeoff.
Also based on the use patterns on our large clusters we read up to 90% of
the data we write, so cold data is a small fraction and most of it must be
cached.
To summarize:
- Ozone is a big enough system to deserve its own project.
- The architecture that Ozone leads to does not seem to solve the intrinsic
problems of current HDFS.
I will post my opinion in the Ozone jira. Should be more convenient to
discuss it there for further reference.
Thanks,
--Konstantin
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confl
> uence/display/HADOOP/Dev+cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>
On Wed, Oct 18, 2017 at 6:54 PM, Yang Weiwei <ch...@hotmail.com> wrote:
> Hello everyone,
>
>
> I would like to start this thread to discuss merging Ozone (HDFS-7240) to
> trunk. This feature implements an object store which can co-exist with
> HDFS. Ozone is disabled by default. We have tested Ozone with cluster sizes
> varying from 1 to 100 data nodes.
>
>
>
> The merge payload includes the following:
>
> 1. All services, management scripts
> 2. Object store APIs, exposed via both REST and RPC
> 3. Master service UIs, command line interfaces
> 4. Pluggable pipeline Integration
> 5. Ozone File System (Hadoop compatible file system implementation,
> passes all FileSystem contract tests)
> 6. Corona - a load generator for Ozone.
> 7. Essential documentation added to Hadoop site.
> 8. Version specific Ozone Documentation, accessible via service UI.
> 9. Docker support for ozone, which enables faster development cycles.
>
>
> To build Ozone and run ozone using docker, please follow instructions in
> this wiki page. https://cwiki.apache.org/confluence/display/HADOOP/Dev+
> cluster+with+docker.
>
>
> We have built a passionate and diverse community to drive this feature
> development. As a team, we have achieved significant progress in past 3
> years since first JIRA for HDFS-7240 was opened on Oct 2014. So far, we
> have resolved almost 400 JIRAs by 20+ contributors/committers from
> different countries and affiliations. We also want to thank the large
> number of community members who were supportive of our efforts and
> contributed ideas and participated in the design of ozone.
>
>
> Please share your thoughts, thanks!
>
>
> -- Weiwei Yang
>