You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Bibin A Chundatt (Jira)" <ji...@apache.org> on 2019/09/23 08:48:00 UTC
[jira] [Comment Edited] (YARN-9627) DelegationTokenRenewer could
block transitionToStandy
[ https://issues.apache.org/jira/browse/YARN-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16935581#comment-16935581 ]
Bibin A Chundatt edited comment on YARN-9627 at 9/23/19 8:47 AM:
-----------------------------------------------------------------
[~manirajv06@gmail.com]
This issue is more like what do we do with renewal request submitted with large number of pending apps.
was (Author: bibinchundatt):
[~manirajv06@gmail.com]
This issue is more like what do we do with renewal submitted if we have lots of pending apps.
> DelegationTokenRenewer could block transitionToStandy
> -----------------------------------------------------
>
> Key: YARN-9627
> URL: https://issues.apache.org/jira/browse/YARN-9627
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: krishna reddy
> Assignee: Bibin A Chundatt
> Priority: Critical
> Attachments: YARN-9627.001.patch, YARN-9627.002.patch, YARN-9627.003.patch
>
>
> Cluster size: 5K
> Running containers: 55K
> *Scenario*: Largenumber of pending applications (around 50K) and performing RM switch over
> Below exception :
> {noformat}
> 2019-06-13 17:39:27,594 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renew Kind: HDFS_DELEGATION_TOKEN, Service: XXXXXXXXX:1616, Ident: (token for root: HDFS_DELEGATION_TOKEN owner=root/hadoop@HADOOP.COM, renewer=yarn, realUser=, issueDate=1560361265181, maxDate=1560966065181, sequenceNumber=104708, masterKeyId=3);exp=1560533965360; apps=[application_1560346941775_20702] in 86397766 ms, appId = [application_1560346941775_20702]
> 2019-06-13 17:39:27,609 WARN org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Unable to add the application to the delegation token renewer on recovery.
> java.lang.NullPointerException
> at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:522)
> at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleDTRenewerAppRecoverEvent(DelegationTokenRenewer.java:953)
> at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:79)
> at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:912)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>
> 2019-06-13 17:58:20,878 ERROR org.apache.zookeeper.ClientCnxn: Time out error occurred for the packet 'clientPath:null serverPath:null finished:false header:: 27,4 replyHeader:: 27,4295687588,0 request:: '/rmstore1/ZKRMStateRoot/RMDTSecretManagerRoot/RMDTMasterKeysRoot/DelegationKey_49,F response:: #31ffffff8a16b74ffffffe129768ffffffdbffffffe949ffffff8dffffffd517ffffffcafffffffa,s{4295423577,4295423577,1560342837789,1560342837789,0,0,0,0,17,0,4295423577} '.
> 2019-06-13 17:58:20,877 INFO org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Renewed delegation-token= [Kind: HDFS_DELEGATION_TOKEN, Service: XXXXXXXXX:1616, Ident: (token for root: HDFS_DELEGATION_TOKEN owner=root/hadoop@HADOOP.COM, renewer=yarn, realUser=, issueDate=1560366110990, maxDate=1560970910990, sequenceNumber=111891, masterKeyId=3);exp=1560534896413; apps=[application_1560346941775_28115]]
> 2019-06-13 17:58:20,924 WARN org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer: Unable to add the application to the delegation token renewer on recovery.
> java.lang.IllegalStateException: Timer already cancelled.
> at java.util.Timer.sched(Timer.java:397)
> at java.util.Timer.schedule(Timer.java:208)
> at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.setTimerForTokenRenewal(DelegationTokenRenewer.java:612)
> at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleAppSubmitEvent(DelegationTokenRenewer.java:523)
> at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.handleDTRenewerAppRecoverEvent(DelegationTokenRenewer.java:953)
> at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer.access$700(DelegationTokenRenewer.java:79)
> at org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$DelegationTokenRenewerRunnable.run(DelegationTokenRenewer.java:912)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org