[jira] [Created] (HDFS-5909) NameNode : Can some features be implemented in a pure YARN app, to synergize broader uniformity for HCFS distributed FS metadata operations?

["jay vyas (JIRA)" <ji...@apache.org>]

jay vyas created HDFS-5909:
------------------------------

             Summary: NameNode : Can some features be implemented in a pure YARN app, to synergize broader uniformity for HCFS distributed FS metadata operations? 
                 Key: HDFS-5909
                 URL: https://issues.apache.org/jira/browse/HDFS-5909
             Project: Hadoop HDFS
          Issue Type: Wish
            Reporter: jay vyas


The HDFS NameNode is a major barrier to HCFS FileSystem compatibility.  Magic that occurs inside of it is :

1) Essential to certain performant applications of hadoop (i.e. hbase).

2) An extension of FileSystem interface properties which isn't necessarily dependant on HDFS.  

For example, in HDFS-5902, the idea of atomic directory swapping comes up.  This can be done in HDFS, but also, maybe in a generic YARN app which provides general distributed FS metadata services to any HCFS provider.

In addition to making NameNode easier to maintain, this would gaurantee broader compatibility of the HCFS ecosystem. 

Now, just brainstorming here, but... Lets take a typical FS operation:  High volume writes.   In a distributed file system, these can cause performance issues when lots of files are created.  YARN itself provides a framework which is generic enough , that we can write a YARN service which, when running, could generically implement certain distributed FS metadata operations.  Then, HCFS providers would be obligated to provide hooks in their API implementations to update the running YARN distributed metadata services.

Obviously implementation details are complex.

But would it be possible to offload some of the complexity of NameNode into a YARN utility which the HCFS community can leverage?

This is a very raw idea, and if its impossible or not practical, I'm open to feedback.  But in the end I think it could be significant innovation both for HDFS (it would make it more maintainable by offloading shared meta data operations), as well as other HCFS tools (because we would be able to share some of the logic that HDFS implements for synchronizing distribtued FS metadata).



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)