[jira] [Created] (IGNITE-8890) Ignite YARN Kerberos - Delegation Token renewal

["Puviarasu (JIRA)" <ji...@apache.org>]

Puviarasu created IGNITE-8890:
---------------------------------

             Summary: Ignite YARN Kerberos - Delegation Token renewal
                 Key: IGNITE-8890
                 URL: https://issues.apache.org/jira/browse/IGNITE-8890
             Project: Ignite
          Issue Type: Bug
          Components: yarn
    Affects Versions: 2.3
         Environment: Kerberos cluster

Ignite Version : 2.3.0
Module : Ignite-YARN
Class : ApplicationMaster

 
            Reporter: Puviarasu


As Ignite-YARN is a long running application in YARN environment it should have a mechanism to renew the delegation token. 

In Ignite-YARN, when the ApplicationMaster is started, it acquires Delegation tokens and stores in a ByteBuffer[Class: ApplicationMaster, Method: init()]. 
This ByteBuffer with token information is given to all the containers received from ResourceManager [Class: ApplicationMaster, Method: onContainersAllocated()]. 
Everything works fine till the life time of the delegation token. 
Once the delegation token expires, the ApplicationMaster is not able to start Ignite inside containers it receive and below exception occurs 

WARNING: Error launching container 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): 
at org.apache.hadoop.ipc.Client.call(Client.java:1504)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2123)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1253)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1249)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1249)
at org.apache.ignite.yarn.utils.IgniteYarnUtils.setupFile(IgniteYarnUtils.java:65)
at org.apache.ignite.yarn.ApplicationMaster.onContainersAllocated(ApplicationMaster.java:131)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)

ApplicationMaster keeps on asking for more and more containers [Class: ApplicationMaster, Method: run()] but not able to start Ignite inside any of the containers due to the expired/missing delegation token. 
This repeats until all the resources in the cluster are allocated to Ignition.

Kindly help in resolving the issue.

Thanks in Advance!!!

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)