You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Shefron Yudy (Jira)" <ji...@apache.org> on 2021/10/26 02:08:00 UTC
[jira] [Commented] (YARN-10516) In HA mode, when one Resource
Manager has networking issue, getTokenService() should not throw runtime
exception
[ https://issues.apache.org/jira/browse/YARN-10516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434078#comment-17434078 ]
Shefron Yudy commented on YARN-10516:
-------------------------------------
I meet it also, the log as follows:
2021-10-21 00:04:13,066 ERROR scheduler.SchedulerApplicationAttempt (SchedulerApplicationAttempt.java:updateContainerAndNMToken(659)) - Error trying to assign container token and NM token to an updated container container_e26_1628079529628_226257_01_0156262021-10-21 00:04:13,066 ERROR scheduler.SchedulerApplicationAttempt (SchedulerApplicationAttempt.java:updateContainerAndNMToken(659)) - Error trying to assign container token and NM token to an updated container container_e26_1628079529628_226257_01_015626java.lang.IllegalArgumentException: java.net.UnknownHostException: hadoop05.example.com at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:445) ~[hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:310) ~[hadoop-yarn-server-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:256) ~[hadoop-yarn-server-resourcemanager-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateContainerAndNMToken(SchedulerApplicationAttempt.java:651) [hadoop-yarn-server-resourcemanager-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.pullNewlyAllocatedContainers(SchedulerApplicationAttempt.java:729) [hadoop-yarn-server-resourcemanager-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.allocate(FairScheduler.java:920) [hadoop-yarn-server-resourcemanager-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor.allocate(DefaultAMSProcessor.java:273) [hadoop-yarn-server-resourcemanager-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain.allocate(AMSProcessingChain.java:92) [hadoop-yarn-server-resourcemanager-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:388) [hadoop-yarn-server-resourcemanager-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60) [hadoop-yarn-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99) [hadoop-yarn-api-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) [hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) [hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869) [hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815) [hadoop-common-3.0.0-cdh6.2.0.jar:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0-internal] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0-internal] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) [hadoop-common-3.0.0-cdh6.2.0.jar:?] at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675) [hadoop-common-3.0.0-cdh6.2.0.jar:?]Caused by: java.net.UnknownHostException: hadoop05.example.com ... 19 more
> In HA mode, when one Resource Manager has networking issue, getTokenService() should not throw runtime exception
> ----------------------------------------------------------------------------------------------------------------
>
> Key: YARN-10516
> URL: https://issues.apache.org/jira/browse/YARN-10516
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client
> Reporter: Xu Cang
> Priority: Minor
> Attachments: YARN-10516.001.patch, YARN-10516.002.patch, YARN-10516.003.patch, YARN-10516.004.patch, YARN-10516.007.patch
>
>
> We have observed one issue from YARN client around this piece of code:
> [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/ClientRMProxy.java#L145]
>
> While
> {code:java}
> services.add(SecurityUtil.buildTokenService( yarnConf.getSocketAddr(address, defaultAddr, defaultPort)) .toString());
>
> {code}
> is being called, buildTokenService() fails and will throw runtime exception, more specifically, UnknownHostException from here: [https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java#L466]
> *while one of the RM host was having networking issue* that IP cannot be resolved.
> This runtime exception then floats all the way up to our application and causes MR job submission failed.
> In my opinion, since we have HA here, multiple RMs are still alive and available. We should catch this exception in getTokenService() and handle it properly, instead of failing the whole action.
>
>
> Would like to hear your opinion on this, if agreed, I will provide a patch on this. Thank you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org