You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2018/05/18 20:05:00 UTC
[jira] [Resolved] (SPARK-24149) Automatic namespaces discovery in
HDFS federation
[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-24149.
------------------------------------
Resolution: Fixed
Fix Version/s: 2.4.0
Issue resolved by pull request 21216
[https://github.com/apache/spark/pull/21216]
> Automatic namespaces discovery in HDFS federation
> -------------------------------------------------
>
> Key: SPARK-24149
> URL: https://issues.apache.org/jira/browse/SPARK-24149
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 2.4.0
> Reporter: Marco Gaido
> Assignee: Marco Gaido
> Priority: Minor
> Fix For: 2.4.0
>
>
> Hadoop 3 introduced HDFS federation.
> Spark fails to write on different namespaces when Hadoop federation is turned on and the cluster is secure. This happens because Spark looks for the delegation token only for the defaultFS configured and not for all the available namespaces. A workaround is the usage of the property {{spark.yarn.access.hadoopFileSystems}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org