[jira] [Created] (BEAM-8399) Python HDFS implementation should support filenames of the format "hdfs://namenodehost/parent/child"

["Chamikara Madhusanka Jayalath (Jira)" <ji...@apache.org>]

Chamikara Madhusanka Jayalath created BEAM-8399:
---------------------------------------------------

             Summary: Python HDFS implementation should support filenames of the format "hdfs://namenodehost/parent/child"
                 Key: BEAM-8399
                 URL: https://issues.apache.org/jira/browse/BEAM-8399
             Project: Beam
          Issue Type: Improvement
          Components: sdk-py-core
            Reporter: Chamikara Madhusanka Jayalath


"hdfs://namenodehost/parent/child" and "/parent/child" seems to be the correct filename formats for HDFS based on [1] but we currently support format "hdfs://parent/child".

To not break existing users, we have to either (1) somehow support both versions by default (based on [2] seems like HDFS does not allow colons in file path so this might be possible) (2) make  "hdfs://namenodehost/parent/child" optional for now and change it to default after few versions.

We should also make sure that Beam Java and Python HDFS file-system implementations are consistent in this regard.

 

[1][https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html]

[2] https://issues.apache.org/jira/browse/HDFS-13

 

cc: [~udim]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)