You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "L. C. Hsieh (Jira)" <ji...@apache.org> on 2021/02/15 07:40:00 UTC

[jira] [Commented] (SPARK-34295) Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude

    [ https://issues.apache.org/jira/browse/SPARK-34295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284588#comment-17284588 ] 

L. C. Hsieh commented on SPARK-34295:
-------------------------------------

To prevent other questioning about the assignee, I have the changed ready locally but I don't have the environment to test. I'd let our customer to test it internally. Once I get the confirmation, I will submit the PR.

> Allow option similar to mapreduce.job.hdfs-servers.token-renewal.exclude
> ------------------------------------------------------------------------
>
>                 Key: SPARK-34295
>                 URL: https://issues.apache.org/jira/browse/SPARK-34295
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 3.2.0
>            Reporter: L. C. Hsieh
>            Assignee: L. C. Hsieh
>            Priority: Major
>
> MapReduce jobs can instruct YARN to skip renewal of tokens obtained from certain hosts by specifying the hosts with configuration mapreduce.job.hdfs-servers.token-renewal.exclude=<host1>,<host2>,..,<hostN>.
> But seems Spark lacks of similar option. So the job submission fails if YARN fails to renew DelegationToken for any of the remote HDFS cluster.  The failure in DT renewal can happen due to many reason like Remote HDFS does not trust Kerberos identity of YARN etc.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org