[jira] [Commented] (ARROW-350) Add Kerberos support to HDFS shim

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612694#comment-15612694 ] 

Wes McKinney commented on ARROW-350:
------------------------------------

There is no reliable support for libhdfs3 -- it was proposed as a patch to Hadoop and rejected. Meanwhile, a new official C++ HDFS client is in the works, see HDFS-8707. If there is a bug we cannot really count on the Apache Hadoop community to fix it. 

Because libhdfs3 is mostly API compatible with libhdfs if someone will do the work we could add an option to switch between libhdfs and libhdfs3 in the HdfsClient factory function. 

> Add Kerberos support to HDFS shim
> ---------------------------------
>
>                 Key: ARROW-350
>                 URL: https://issues.apache.org/jira/browse/ARROW-350
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++, Python
>            Reporter: Christopher Aycock
>            Assignee: Christopher Aycock
>
> Speaking with [~wesmckinn], it would be really helpful to have Kerberos support in our HDFS logic. This should be straightforward; I would just need to switch to {{hdfsBuilderConnect()}} in the shim.
> On a side note, is there a reason we aren't using Pivotal's libhdfs3? It uses RPCs natively rather than JNI.
> https://github.com/Pivotal-Data-Attic/pivotalrd-libhdfs3
> Dask has Python wrappers for this.
> https://github.com/dask/hdfs3



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)