You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/08/08 02:49:11 UTC

[jira] [Commented] (HDFS-2656) Implement a pure c client based on webhdfs

    [ https://issues.apache.org/jira/browse/HDFS-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430778#comment-13430778 ] 

Todd Lipcon commented on HDFS-2656:
-----------------------------------

Can you please comment on performance of this implementation vs the libhdfs in trunk?

I am uncomfortable calling this "libhdfs2" if the performance is not up to par with the current "libhdfs1". One of the major reasons people use libhdfs is that they're writing a high-performance app, and I would be really surprised if this HTTP-based client can match the performance of the JNI-based one, especially when short-circuit read is enabled.

An additional concern is that webhdfs doesn't seem to return any checksum information with its transfers. So, a client using this C library no longer has the benefits of checksumming that HDFS provides.

So while I see there are some uses for this, I don't think it's viable as a _replacement_. Perhaps better to describe it as "libwebhdfs" or something?

The "libhdfs3" work on the other hand sounds useful for a variety of applications, and should have better performance and memory footprint while not giving up features like checksumming, etc. Any word on when that would be available?
                
> Implement a pure c client based on webhdfs
> ------------------------------------------
>
>                 Key: HDFS-2656
>                 URL: https://issues.apache.org/jira/browse/HDFS-2656
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: webhdfs
>            Reporter: Zhanwei.Wang
>         Attachments: HDFS-2656.patch, HDFS-2656.unfinished.patch
>
>
> Currently, the implementation of libhdfs is based on JNI. The overhead of JVM seems a little big, and libhdfs can also not be used in the environment without hdfs.
> It seems a good idea to implement a pure c client by wrapping webhdfs. It also can be used to access different version of hdfs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira