You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "Fabian Höring (Jira)" <ji...@apache.org> on 2019/12/04 10:36:00 UTC

[jira] [Created] (ARROW-7309) [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path

Fabian Höring created ARROW-7309:
------------------------------------

             Summary: [Python] Support HDFS federation viewfs:// in resolve_filesystem_and_path
                 Key: ARROW-7309
                 URL: https://issues.apache.org/jira/browse/ARROW-7309
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Python
    Affects Versions: 0.15.1
            Reporter: Fabian Höring


- Add viewfs support to pyarrow.filesystem.resolve_filesystem_and_path
-  libhdfs already supports injecting the scheme and will automatically resolve federation in
    fs = FileSystem#get(URI, conf, ugi)
-  works with Hadoop 2/3

see:
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L770
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c#L637



--
This message was sent by Atlassian Jira
(v8.3.4#803005)