You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peter Parente (JIRA)" <ji...@apache.org> on 2017/01/06 12:40:58 UTC
[jira] [Created] (SPARK-19105) yarn/Client.scala copyToRemote does
not include keytab destination name
Peter Parente created SPARK-19105:
-------------------------------------
Summary: yarn/Client.scala copyToRemote does not include keytab destination name
Key: SPARK-19105
URL: https://issues.apache.org/jira/browse/SPARK-19105
Project: Spark
Issue Type: Bug
Affects Versions: 2.1.0
Environment: YARN in client mode
Reporter: Peter Parente
When I specify {{monospaced}}--principal user@REALM{{monospaced}} and {{monospaced}}--keytab /some/path/user.keytab{{monospaced}}, I see the following in my app staging directory on HDFS:
{code}
-rw-r--r-- 3 user supergroup 68 2017-01-06 03:59 user.keytab
-rw-r--r-- 3 user supergroup 73502 2017-01-06 03:59 __spark_conf__.zip
-rw-r--r-- 3 user supergroup 189767340 2017-01-06 03:59 __spark_libs__4440821503780683972.zip
-rw-r--r-- 3 user supergroup 91275 2017-01-06 03:59 py4j-0.10.3-src.zip
-rw-r--r-- 3 user supergroup 440385 2017-01-06 03:59 pyspark.zip
{code}
I also see that my spark.yarn.keytab config value has changed to user.keytab-54ee5192-43d0-41b5-ba50-1181ece26961 by the yarn client to ensure the keytab is unique within the app staging directory. However, from the directory listing above, it's clear that the file written does not match this new name. As a result, when it comes time to renew the Kerberos ticket, AMDelegationTokenRenewer fails to find the keytab under the UUID-suffixed name and also fails to renew the tickets.
The problem looks to be in one call to [copyFileToRemote in yarn/Client.java|https://github.com/apache/spark/blob/fe1c895e16c475a6f271ce600a42a8d0dc7986e5/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L482] that leaves off the destination filename param. The other calls in that object which use copyFileToRemote and have a custom destination name all provide this parameter (e.g., https://github.com/apache/spark/blob/fe1c895e16c475a6f271ce600a42a8d0dc7986e5/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L652).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org