You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jing Zhao (JIRA)" <ji...@apache.org> on 2013/10/08 01:00:42 UTC
[jira] [Commented] (HDFS-5322) HDFS delegation token not found in
cache errors seen on secure HA clusters
[ https://issues.apache.org/jira/browse/HDFS-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13788672#comment-13788672 ]
Jing Zhao commented on HDFS-5322:
---------------------------------
Looks like the issue happened during the standby->active transitioning process of the namenode. The namenode had not finished tailing the editlog, which contained information about generating the corresponding token, thus failed to find the token in its local cache.
A possible solution is to let the client side retry the same operation in this scenario. Thus if the namenode could not find the token during the transition, it can wrap the InvalidToken exception in the RetriableException and ask the client to retry.
> HDFS delegation token not found in cache errors seen on secure HA clusters
> --------------------------------------------------------------------------
>
> Key: HDFS-5322
> URL: https://issues.apache.org/jira/browse/HDFS-5322
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
> Affects Versions: 2.1.1-beta
> Reporter: Arpit Gupta
> Assignee: Jing Zhao
>
> While running HA tests we have seen issues were we see HDFS delegation token not found in cache errors causing jobs running to fail.
> {code}
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> |2013-10-06 20:14:51,193 INFO [main] mapreduce.Job: Task Id : attempt_1381090351344_0001_m_000007_0, Status : FAILED
> Error: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): token (HDFS_DELEGATION_TOKEN token 11 for hrt_qa) can't be found in cache
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy10.getBlockLocations(Unknown Source)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)