[jira] [Commented] (HADOOP-10388) Pure native hadoop client

["Binglin Chang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-10388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13956157#comment-13956157 ] 

Binglin Chang commented on HADOOP-10388:
----------------------------------------

bq. We can even make the XML-reading code optional if you want.
Sure, if for compatibility I guess add xml support if fine. By keeping strict compatibility we may need to support all javax xml / hadoop config features, I'm afraid libexpact/libxml2 support all of those, a lot effort may be spent on this, it is better to make it optional and do it later I think.

bq. Thread pools and async I/O, I'm afraid, are something we can't live without.
I am also prefer to use async I/O and thread for performance reasons, the code I published on github already have a working HDFS client with read/write, and HDFSOuputstream uses an aditional thread. 
What I was saying is use of more threads should be limited, in java client, to simply read/write a HDFS file, too much threads are used(rpc socket read/write, data transfer socket read/write, other misc executors, lease renewer etc.) Since we use async i/o, thread number should be rapidly reduced


> Pure native hadoop client
> -------------------------
>
>                 Key: HADOOP-10388
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10388
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: HADOOP-10388
>            Reporter: Binglin Chang
>            Assignee: Colin Patrick McCabe
>
> A pure native hadoop client has following use case/advantages:
> 1.  writing Yarn applications using c++
> 2.  direct access to HDFS, without extra proxy overhead, comparing to web/nfs interface.
> 3.  wrap native library to support more languages, e.g. python
> 4.  lightweight, small footprint compare to several hundred MB of JDK and hadoop library with various dependencies.



--
This message was sent by Atlassian JIRA
(v6.2#6252)