You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Raghav <pu...@gmail.com> on 2018/06/27 12:12:15 UTC
Ignite YARN Kerberos - Delegation Ticket renewal
Ignite YARN not having a facility to renew HDFS_DELEGATION_TOKEN.
As Ignite-YARN is a long running application in YARN environment it should
have a mechanism to renew the delegation token.
Version : 2.3.0
Module : Ignite-YARN
Class : ApplicationMaster
In Ignite-YARN, when the ApplicationMaster is started, it acquires
Delegation tokens and stores in a ByteBuffer[Class: ApplicationMaster,
Method: init()].
This ByteBuffer with token information is given to all the containers
received from ResourceManager [Class: ApplicationMaster, Method:
onContainersAllocated()].
Everything works fine till the life time of the delegation token.
Once the delegation token expires, the ApplicationMaster is not able to
start Ignite inside containers it receive and below exception occurs
WARNING: Error launching container
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
at org.apache.hadoop.ipc.Client.call(Client.java:1504)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:771)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2123)
at
org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1253)
at
org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1249)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1249)
at
org.apache.ignite.yarn.utils.IgniteYarnUtils.setupFile(IgniteYarnUtils.java:65)
at
org.apache.ignite.yarn.ApplicationMaster.onContainersAllocated(ApplicationMaster.java:131)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:292)
ApplicationMaster keeps on asking for more and more containers [Class:
ApplicationMaster, Method: run()] but not able to start Ignite inside any of
the containers due to the expired/missing delegation token.
This repeats until all the resources in the cluster are allocated to
Ignition.
Kindly help in resolving the issue.
Thanks in Advance!!!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite YARN Kerberos - Delegation Ticket renewal
Posted by Denis Mekhanikov <dm...@gmail.com>.
Thank you for the report!
Please consider sending a pull request.
The community will appreciate your YARN expertise.
Denis
ср, 27 июн. 2018 г. в 16:15, Raghav <pu...@gmail.com>:
> Hello together,
>
> I have also logged this issue in issues.apache.org.
>
> Kindly requesting your support.
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite YARN Kerberos - Delegation Ticket renewal
Posted by Raghav <pu...@gmail.com>.
Hello together,
I have also logged this issue in issues.apache.org.
Kindly requesting your support.
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/