You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Andras Piros (JIRA)" <ji...@apache.org> on 2017/08/14 15:12:00 UTC
[jira] [Created] (OOZIE-3036) Spark 2.2.0 support: tell Spark not
to get any delegation tokens
Andras Piros created OOZIE-3036:
-----------------------------------
Summary: Spark 2.2.0 support: tell Spark not to get any delegation tokens
Key: OOZIE-3036
URL: https://issues.apache.org/jira/browse/OOZIE-3036
Project: Oozie
Issue Type: Bug
Affects Versions: 4.3.0
Environment: * [*Spark 2.2.0*|https://github.com/apache/spark/tree/branch-2.2]
* [*Hadoop 3 alpha 4*|https://github.com/apache/hadoop/tree/branch-3.0.0-alpha4]
Reporter: Andras Piros
Assignee: Andras Piros
Fix For: 5.0.0
For Spark 2.2.0 along w/ Hadoop 3 alpha 4, Spark uses another set of arguments that ensure Spark won't get any delegation tokens:
* {{spark.yarn.security.credentials.hadoopfs.enabled}}
* {{spark.yarn.security.credentials.hive.enabled}}
* {{spark.yarn.security.credentials.hbase.enabled}}
For Spark 1.6.0 along w/ Hadoop 3 alpha 4, we need following argument to set extra:
* {{spark.yarn.security.tokens.hadoopfs.enabled}}
As we can't tell runtime which Spark / Hadoop versions is the Oozie Spark action running on, it's best to use the union of those arguments.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)