You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by "Xie YiFan (Jira)" <ji...@apache.org> on 2019/09/04 10:32:00 UTC
[jira] [Created] (YARN-9811) FederationInterceptor fails to recover
in Kerberos environment
Xie YiFan created YARN-9811:
-------------------------------
Summary: FederationInterceptor fails to recover in Kerberos environment
Key: YARN-9811
URL: https://issues.apache.org/jira/browse/YARN-9811
Project: Hadoop YARN
Issue Type: Bug
Components: amrmproxy
Reporter: Xie YiFan
Assignee: Xie YiFan
*scenario*:
Start up cluster in Kerberos environment with enable recover & AMRMProxy in NM. Submit one application to cluster, and restart NM which has master container. The NM will block in FederationInterceptor recover.
*LOG*
{code:java}
INFO org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor: Recovering data for FederationInterceptor
INFO org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor: Found 0 existing UAMs for application application_1561534175896_4102 in NMStateStore
INFO org.apache.hadoop.yarn.server.utils.AMRMClientUtils: Creating RMProxy to RM online-bx for protocol ApplicationClientProtocol for user recommend (auth:SIMPLE)
INFO org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider: Initialized Federation proxy for user: recommend
INFO org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider: Failing over to the ResourceManager for SubClusterId: online-bx
INFO org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider: Connecting to /10.88.86.142:8032 subClusterId online-bx with protocol ApplicationClientProtocol as user recommend (auth:SIMPLE)
WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
INFO org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider: Failing over to the ResourceManager for SubClusterId: online-bx
INFO org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade: Flushing subClusters from cache and rehydrating from store, most likely on account of RM failover.
INFO org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider: Connecting to /10.88.86.142:8032 subClusterId online-bx with protocol ApplicationClientProtocol as user recommend (auth:SIMPLE)
WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
INFO org.apache.hadoop.io.retry.RetryInvocationHandler: java.io.IOException: DestHost:destPort hadoop1684.bx.momo.com:8032 , LocalHost:localPort hadoop999.bx.momo.com/10.88.64.186:0. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS], while invoking ApplicationClientProtocolPBClientImpl.getContainers over online-bx after 1 failover attempts. Trying to failover after sleeping for 3244ms.{code}
*Analysis*
rmclient.getContainers is called. But AuthMethod of appSubmitter is SIMPLE.We should use createProxyUser instead of createRemoteUser in Security.
{code:java}
UserGroupInformation appSubmitter = UserGroupInformation .createRemoteUser(getApplicationContext().getUser()); ApplicationClientProtocol rmClient = createHomeRMProxy(getApplicationContext(), ApplicationClientProtocol.class, appSubmitter);
GetContainersResponse response = rmClient .getContainers(GetContainersRequest.newInstance(this.attemptId));
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org