You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "krishna reddy (JIRA)" <ji...@apache.org> on 2019/06/14 13:58:00 UTC
[jira] [Created] (YARN-9627) DelegationTokenRenewer throws
exception on switchover
krishna reddy created YARN-9627:
-----------------------------------
Summary: DelegationTokenRenewer throws exception on switchover
Key: YARN-9627
URL: https://issues.apache.org/jira/browse/YARN-9627
Project: Hadoop YARN
Issue Type: Bug
Reporter: krishna reddy
Cluster size: 5K
Running containers: 55K
*Scenario*: Largenumber of pending applications (around 50K) and performing RM switch over
Below exception :
{noformat}
2019-06-13 17:39:27,594 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: HDFS_DELEGATION_TOKEN, Service: XXXXXXXXX:1616, Ident: (token for root: HDFS_DELEGATION_TOKEN owner=root/hadoop@HADOOP.COM, renewer=yarn, realUser=, issueDate=1560361265181, maxDate=1560966065181, sequenceNumber=104708, masterKeyId=3);exp=1560533965360; apps=[application_1560346941775_20702] in 86397766 ms, appId = [application_1560346941775_20702]
2019-06-13 17:39:27,609 WARN org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Unable to add the application to the delegation token renewer on recovery.
java.lang.NullPointerException
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:522)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleDTRenewerAppRecoverEvent(DelegationTokenRenewer.java:953)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:79)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:912)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-06-13 17:58:20,878 ERROR org.apache.zookeeper.ClientCnxn: Time out error occurred for the packet 'clientPath:null serverPath:null finished:false header:: 27,4 replyHeader:: 27,4295687588,0 request:: '/rmstore1/ZKRMStateRoot/RMDTSecretManagerRoot/RMDTMasterKeysRoot/DelegationKey_49,F response:: #31ffffff8a16b74ffffffe129768ffffffdbffffffe949ffffff8dffffffd517ffffffcafffffffa,s{4295423577,4295423577,1560342837789,1560342837789,0,0,0,0,17,0,4295423577} '.
2019-06-13 17:58:20,877 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: HDFS_DELEGATION_TOKEN, Service: XXXXXXXXX:1616, Ident: (token for root: HDFS_DELEGATION_TOKEN owner=root/hadoop@HADOOP.COM, renewer=yarn, realUser=, issueDate=1560366110990, maxDate=1560970910990, sequenceNumber=111891, masterKeyId=3);exp=1560534896413; apps=[application_1560346941775_28115]]
2019-06-13 17:58:20,924 WARN org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Unable to add the application to the delegation token renewer on recovery.
java.lang.IllegalStateException: Timer already cancelled.
at java.util.Timer.sched(Timer.java:397)
at java.util.Timer.schedule(Timer.java:208)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:612)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:523)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleDTRenewerAppRecoverEvent(DelegationTokenRenewer.java:953)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:79)
at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:912)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org