[jira] [Commented] (YARN-10601) The Yarn client should use the UGI who created the Yarn client for obtaining a delegation token for the remote log dir

["Prabhu Joseph (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-10601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276106#comment-17276106 ] 

Prabhu Joseph commented on YARN-10601:
--------------------------------------

[~fritsi] Thanks for the details.

>> As you can see submitApplication is not invoked inside an ugi.doAs block

Why submitApplication is not invoked inside ugi.doAs block. If we need the log aggregation to happen as per submitterUser, the job also has to be submitted by submitterUser right?

> The Yarn client should use the UGI who created the Yarn client for obtaining a delegation token for the remote log dir
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-10601
>                 URL: https://issues.apache.org/jira/browse/YARN-10601
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: log-aggregation
>    Affects Versions: 3.3.0, 3.4.0
>            Reporter: Daniel Fritsi
>            Priority: Critical
>
> It seems there was a bug introduced in YARN-10333 in this section of *{color:#0747A6}{{addLogAggregationDelegationToken}}{color}*:
> {code:java}
> Path remoteRootLogDir = fileController.getRemoteRootLogDir();
> FileSystem fs = remoteRootLogDir.getFileSystem(conf);
> final org.apache.hadoop.security.token.Token<?>[] finalTokens =
>     fs.addDelegationTokens(masterPrincipal, credentials);
> {code}
> *{color:#0747A6}{{remoteRootLogDir.getFileSystem}}{color}* simply does this:
> {code:java}
> public FileSystem getFileSystem(Configuration conf) throws IOException {
>   return FileSystem.get(this.toUri(), conf);
> }
> {code}
> As far as I know it's customary to create a YarnClient instance via *{color:#0747A6}{{YarnClient.createYarnClient()}}{color}* in a UserGroupInformation.doAs block if you would like to use it with a different user then the current one. E.g.:
> {code:java}
> YarnClient yarnClient = ugi.doAs(new PrivilegedExceptionAction<YarnClient>() {
>     @Override
>     public YarnClient run() throws Exception {
>         YarnClient yarnClient = YarnClient.createYarnClient();
>         yarnClient.init(conf);
>         yarnClient.start();
>         return yarnClient;
>     }
> });
> {code}
> If this statement is correct then I think YarnClient should save the *{color:#0747A6}{{UserGroupInformation.getCurrentUser()}}{color}* when the YarnClient is being created and the *{color:#0747A6}{{remoteRootLogDir.getFileSystem(conf)}}{color}* call should be made inside an ugi.doAs block with that saved user.
> A more concrete example:
> {code:java}
> public YarnClient createYarnClient(UserGroupInformation ugi, Configuration conf) throws Exception {
>     return ugi.doAs((PrivilegedExceptionAction<YarnClient>) () -> {
>         // Her I am the submitterUser (see below)
>         YarnClient yarnClient = YarnClient.createYarnClient();
>         yarnClient.init(conf);
>         yarnClient.start();
>         return yarnClient;
>     });
> }
> public void run() {
>     // Here I am the serviceUser
>     // ...
>     Configuration conf = ...
>     // ...
>     UserGroupInformation ugi = getSubmitterUser();
>     // ...
>     YarnClient yarnClient = createYarnClient(ugi);
>     // ...
>     ApplicationSubmissionContext context = ...
>     // ...
>     yarnClient.submitApplication(context);
> }
> {code}
> As you can see *{color:#0747A6}{{submitApplication}}{color}* is not invoked inside an ugi.doAs block and submitApplication is the one who will eventually invoke *{color:#0747A6}{{addLogAggregationDelegationToken}}{color}*. That's why we need to save the UGI during the YarnClient creation and create the FileSystem instance inside an ugi.doAs with that saved user. Otherwise Yarn will try to get a delegation token with an incorrect user (serviceUser) instead of the submitterUser.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org