[jira] [Comment Edited] (OOZIE-2909) LauncherAM: rewrite UGI calls

["Peter Bacsko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-2909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153291#comment-16153291 ] 

Peter Bacsko edited comment on OOZIE-2909 at 9/5/17 9:26 AM:
-------------------------------------------------------------

Relevant code we need:

https://github.com/apache/hadoop/blob/3721cfe1fbd98c5b6aa46aefdfcf62276c28c4a4/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L236-L245
{code}
  protected MRClientProtocol instantiateHistoryProxy()
      throws IOException {
    final String serviceAddr = conf.get(JHAdminConfig.MR_HISTORY_ADDRESS);
    if (StringUtils.isEmpty(serviceAddr)) {
      return null;
    }
    LOG.debug("Connecting to HistoryServer at: " + serviceAddr);
    final YarnRPC rpc = YarnRPC.create(conf);
    LOG.debug("Connected to HistoryServer at: " + serviceAddr);
    UserGroupInformation currentUser = UserGroupInformation.getCurrentUser();
    return currentUser.doAs(new PrivilegedAction<MRClientProtocol>() {
      @Override
      public MRClientProtocol run() {
        return (MRClientProtocol) rpc.getProxy(HSClientProtocol.class,
            NetUtils.createSocketAddr(serviceAddr), conf);
      }
    });
  }
{code}

https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java#L82-L99
{code}
@VisibleForTesting
  Token<?> getDelegationTokenFromHS(MRClientProtocol hsProxy)
      throws IOException, InterruptedException {
    GetDelegationTokenRequest request = recordFactory
      .newRecordInstance(GetDelegationTokenRequest.class);
    request.setRenewer(Master.getMasterPrincipal(conf));
    org.apache.hadoop.yarn.api.records.Token mrDelegationToken;
    mrDelegationToken = hsProxy.getDelegationToken(request)
        .getDelegationToken();
    return ConverterUtils.convertFromYarn(mrDelegationToken,
        hsProxy.getConnectAddress());
  }
{code}

Not the nicest code, but we have to try it. Only questionable part is the usage of {{YarnRPC}}, which is marked as {{LimitedPrivate}}, but perhaps it's still acceptable.


was (Author: pbacsko):
Relevant code:

{code}
  protected MRClientProtocol instantiateHistoryProxy()
      throws IOException {
    final String serviceAddr = conf.get(JHAdminConfig.MR_HISTORY_ADDRESS);
    if (StringUtils.isEmpty(serviceAddr)) {
      return null;
    }
    LOG.debug("Connecting to HistoryServer at: " + serviceAddr);
    final YarnRPC rpc = YarnRPC.create(conf);
    LOG.debug("Connected to HistoryServer at: " + serviceAddr);
    UserGroupInformation currentUser = UserGroupInformation.getCurrentUser();
    return currentUser.doAs(new PrivilegedAction<MRClientProtocol>() {
      @Override
      public MRClientProtocol run() {
        return (MRClientProtocol) rpc.getProxy(HSClientProtocol.class,
            NetUtils.createSocketAddr(serviceAddr), conf);
      }
    });
  }
{code}

{code}
@VisibleForTesting
  Token<?> getDelegationTokenFromHS(MRClientProtocol hsProxy)
      throws IOException, InterruptedException {
    GetDelegationTokenRequest request = recordFactory
      .newRecordInstance(GetDelegationTokenRequest.class);
    request.setRenewer(Master.getMasterPrincipal(conf));
    org.apache.hadoop.yarn.api.records.Token mrDelegationToken;
    mrDelegationToken = hsProxy.getDelegationToken(request)
        .getDelegationToken();
    return ConverterUtils.convertFromYarn(mrDelegationToken,
        hsProxy.getConnectAddress());
  }
{code}

Not the nicest code, but we have to try it. Only questionable part is the usage of {{YarnRPC}}, which is marked as {{LimitedPrivate}}, but perhaps it's still acceptable.

> LauncherAM: rewrite UGI calls
> -----------------------------
>
>                 Key: OOZIE-2909
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2909
>             Project: Oozie
>          Issue Type: Sub-task
>            Reporter: Peter Bacsko
>            Assignee: Peter Cseh
>            Priority: Blocker
>         Attachments: OOZIE-2909-001.patch
>
>
> There are two problems in LauncherAM:
> 1) In a kerberized cluster, we have to use {{UGI.getLoginUser()}} even though a call to {{createRemoteUser()}} should be enough, just like in {{MRAppMaster.java}}. Reference code:
> https://github.com/apache/hadoop/blob/3ed3062fe3979ff55a411b730a8eee2b2c96d6b3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java#L1718-L1726
> 2) It's enough if we wrap the whole LauncherAM logic in a single {{doAs()}} call. No need for multiple {{doAs()}}, just makes the whole thing more complicated.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)