You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2019/02/12 20:45:00 UTC
[jira] [Resolved] (SPARK-11662) Call
startExecutorDelegationTokenRenewer() ahead of client app submission
[ https://issues.apache.org/jira/browse/SPARK-11662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Marcelo Vanzin resolved SPARK-11662.
------------------------------------
Resolution: Cannot Reproduce
Pretty sure this works these days.
> Call startExecutorDelegationTokenRenewer() ahead of client app submission
> -------------------------------------------------------------------------
>
> Key: SPARK-11662
> URL: https://issues.apache.org/jira/browse/SPARK-11662
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Reporter: Ted Yu
> Priority: Major
>
> As reported in the thread 'Creating new Spark context when running in Secure YARN fails', IOException may be thrown when SparkContext is stopped and started again working with secure YARN cluster:
> {code}
> 15/11/11 10:19:53 ERROR spark.SparkContext: Error initializing SparkContext.
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): Delegation Token
> can be issued only with kerberos or web authentication
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getDelegationToken(FSNamesystem.java:6638)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getDelegationToken(NameNodeRpcServer.java:563)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getDelegationToken(ClientNamenodeProtocolServerSideTranslatorPB.java:987)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> at org.apache.hadoop.ipc.Client.call(Client.java:1476)
> at org.apache.hadoop.ipc.Client.call(Client.java:1407)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy12.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken(ClientNamenodeProtocolTranslatorPB.java:933)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy13.getDelegationToken(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSClient.getDelegationToken(DFSClient.java:1044)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(DistributedFileSystem.java:1543)
> at
> org.apache.hadoop.fs.FileSystem.collectDelegationTokens(FileSystem.java:530)
> at
> org.apache.hadoop.fs.FileSystem.addDelegationTokens(FileSystem.java:508)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2228)
> at
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:126)
> at
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$$anonfun$obtainTokensForNamenodes$1.apply(YarnSparkHadoopUtil.scala:123)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
> at
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil.obtainTokensForNamenodes(YarnSparkHadoopUtil.scala:123)
> at
> org.apache.spark.deploy.yarn.Client.getTokenRenewalInterval(Client.scala:495)
> at
> org.apache.spark.deploy.yarn.Client.setupLaunchEnv(Client.scala:528)
> at
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:628)
> at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:119)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:56)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:523)
> {code}
> One fix is to call startExecutorDelegationTokenRenewer(conf) ahead of client app submission.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org