[jira] [Commented] (HADOOP-10640) Implement Namenode RPCs in HDFS native client

["Colin Patrick McCabe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-10640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14012839#comment-14012839 ] 

Colin Patrick McCabe commented on HADOOP-10640:
-----------------------------------------------

With this patch, libhdfs.so continues to implement the same API as before.  At runtime, we select which native filesystem implementation to use based on by looking at {{<scheme>.native.handler}}.  So, if the scheme of the URI we're handling is HDFS, we'd look at {{hdfs.native.handler}}.  If there is no scheme-specific handler, we fall back to {{default.native.handler}}.

There are two libhdfs native filesystems available at the moment.  One is jniFS, the existing JNI-based filesystem.  The other is NDFS, the new native filesystem that we're developing.  As mentioned earlier, jniFS has the advantage of working with things like s3 or HDFS encryption, whereas NDFS has the advantage that it doesn't require JNI, is more efficient, etc.  The goal here is to have a {{libhdfs.so}} that we can drop into existing installations and have everything "just work."

The libhdfs.so library doesn't depend on JNI... it can be installed in cases where JNI is not present.  The reason is because it uses {{dlopen}} (by means of the libuv shims) to open the {{libjvm.so}} library as-needed.  So clients who don't want to install or use JNI don't have to.

The build changed a little bit to have a {{config.h.cmake}} file which gets transformed into {{config.h}} based on which features are present at compile-time.  This is how the existing libhdfs build works (we need it to detect some platform-specific features.)  I also hide all library symbols except for the public symbols, so that our library internals are not visible to the outside world, using {{fvisibility=hidden}}.

source overview:
* common/hadoop_err: added the {{hadoop_err_prepend}}, {{hadoop_err_copy}}, and {{terror}} utility functions.
* common/hconf: this is a native implementation of the Java Configuration class which reads XML files.  It uses libexpat.
* common/htable: a hash table used in a few places.
* common/net: added {{net_ipv4_name_and_port}}, {{get_first_ipv4_addr}} utility functions
* common/string: added the {{strdupto}} utility function plus a unit test.
* common/uri: parses HDFS-style URIs into scheme, authority, port, etc. using the uriparser library.
* fs/fs.c: the entry points for all libhdfs functions.  Contains a vtable used to call the appropriate implementation function.
* fs/hdfs.h: the public API of libhdfs.  This is just a rename.
* jni/: all the stuff in this directory was moved from the previous libhdfs location.
* net/proxy, net/reactor: fixed a bug in the synchronous RPC code
* test/: I created a test directory paralleling the source directory.  Moved test.h to this directory.

Running {{NAMENODE_URI=hdfs://localhost:6000 ./test_libhdfs_meta_ops}} will give an idea of how this operates.

> Implement Namenode RPCs in HDFS native client
> ---------------------------------------------
>
>                 Key: HADOOP-10640
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10640
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: native
>    Affects Versions: HADOOP-10388
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>
> Implement the parts of libhdfs that just involve making RPCs to the Namenode, such as mkdir, rename, etc.



--
This message was sent by Atlassian JIRA
(v6.2#6252)