[jira] [Created] (YARN-3804) Both RM are on standBy state when kerberos user not in yarn.admin.acl

["Bibin A Chundatt (JIRA)" <ji...@apache.org>]

Bibin A Chundatt created YARN-3804:
--------------------------------------

             Summary: Both RM are on standBy state when kerberos user not in yarn.admin.acl
                 Key: YARN-3804
                 URL: https://issues.apache.org/jira/browse/YARN-3804
             Project: Hadoop YARN
          Issue Type: Bug
          Components: resourcemanager
         Environment: Suse 11 Sp3, 2 RM, Secure
            Reporter: Bibin A Chundatt


Steps to reproduce
================
1. Configure cluster in secure mode
2. On  RM Configure yarn.admin.acl=dsperf
3. Configure in arn.resourcemanager.principal=yarn
4. Start Both RM 

Both RM will be in Standby forever

{code}

2015-06-15 12:20:21,556 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=yarn     OPERATION=refreshAdminAcls      TARGET=AdminService     RESULT=FAILURE  DESCRIPTION=Unauthorized userPERMISSIONS=
2015-06-15 12:20:21,556 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
        at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:128)
        at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:824)
        at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:420)
        at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:645)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:518)
Caused by: org.apache.hadoop.ha.ServiceFailedException: Can not execute refreshAdminAcls
        at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:297)
        at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:126)
        ... 4 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: org.apache.hadoop.security.AccessControlException: User yarn doesn't have permission to call 'refreshAdminAcls'
        at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
        at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAcls(AdminService.java:230)
        at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAdminAcls(AdminService.java:465)
        at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:295)
        ... 5 more
Caused by: org.apache.hadoop.security.AccessControlException: User yarn doesn't have permission to call 'refreshAdminAcls'
        at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:182)
        at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.verifyAdminAccess(RMServerUtils.java:148)
        at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAccess(AdminService.java:223)
        at org.apache.hadoop.yarn.server.resourcemanager.AdminService.checkAcls(AdminService.java:228)
        ... 7 more
{code}



*Analysis*

On each RM attempt to switch to Active refreshACl is called and acl permission not available for the user
Infinite retry for the same switch to Active and always false returned from 
{{ActiveStandbyElector#becomeActive()}}
 

*Expected*

RM should get shutdown event after few retry or even at first attempt
Since at runtime user from which it retries for refreshacl can never be updated.

*States from commands*

 ./yarn rmadmin -getServiceState rm2
*standby*
 ./yarn rmadmin -getServiceState rm1
*standby*

 ./yarn rmadmin -checkHealth rm1
*echo $? = 0*
 ./yarn rmadmin -checkHealth rm2
*echo $? = 0*




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)