You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Marco Gaido (JIRA)" <ji...@apache.org> on 2018/05/02 14:01:00 UTC

[jira] [Created] (SPARK-24149) Automatic namespaces discovery in HDFS federation

Marco Gaido created SPARK-24149:
-----------------------------------

             Summary: Automatic namespaces discovery in HDFS federation
                 Key: SPARK-24149
                 URL: https://issues.apache.org/jira/browse/SPARK-24149
             Project: Spark
          Issue Type: Improvement
          Components: YARN
    Affects Versions: 2.4.0
            Reporter: Marco Gaido


Hadoop 3 introduced HDFS federation.
Spark fails to write on different namespaces when Hadoop federation is turned on and the cluster is secure. This happens because Spark looks for the delegation token only for the defaultFS configured and not for all the available namespaces. A workaround is the usage of the property {{spark.yarn.access.hadoopFileSystems}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org