You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@reef.apache.org by "Rogan Carr (JIRA)" <ji...@apache.org> on 2017/09/24 05:56:00 UTC

[jira] [Created] (REEF-1892) HDFS File Copy only uses local HDFS

Rogan Carr created REEF-1892:
--------------------------------

             Summary: HDFS File Copy only uses local HDFS
                 Key: REEF-1892
                 URL: https://issues.apache.org/jira/browse/REEF-1892
             Project: REEF
          Issue Type: Bug
          Components: REEF.NET IO
    Affects Versions: 0.17
            Reporter: Rogan Carr


In REEF-1827 [1], the URI used to specify remote and local files were changed to use the "AbsolutePath". [2]
This means that a file specified as "hdfs://my/file" becomes "/my/file" and the hdfs:// is assumed by the `dfs` command.

This is fine if you are using vanilla HDFS, but for cases like Blob Storage in Azure, there is a special prefix, `wasb://` that is used instead of `hdfs://`. This means that the AbsolutePath method trims off the wasb, and this Copy() function instead attempts to download the file from the local HDFS instead of WASB.

We need to revisit this issue and keep the full path for copies while also keeping proper casing in the path.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)