[jira] [Commented] (TEZ-2886) Ability to merge AM credentials with DAG credentials

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955549#comment-14955549 ] 

Jason Lowe commented on TEZ-2886:
---------------------------------

For example, the RM will automatically add tokens for the log aggregation filesystem if they are missing from the job's credentials.  We have some clusters setup where the log aggregation filesystem is not the default filesystem, and the RM is automatically adding the token for the remote log aggregation filesystem.  In this case Tez logs for any tasks are not aggregated because only the AM container has the necessary log aggregation filesystem token.  Other containers only have the DAG credentials, and since the client was unaware of the log aggregation filesystem it doesn't pass it as part of the DAG credentials.

One could argue that the client should look to see if the log aggregation filesystem is not the default filesystem, but clients don't necessarily have the correct setting for those configs (it's a YARN server config and not a client-side config).  Therefore it's not appropriate for the client to try to work around this.

This may be related to TEZ-1588 since I see that patch was also proposing to merge the AM credentials with the DAG credentials in the DAGAppMaster.


> Ability to merge AM credentials with DAG credentials
> ----------------------------------------------------
>
>                 Key: TEZ-2886
>                 URL: https://issues.apache.org/jira/browse/TEZ-2886
>             Project: Apache Tez
>          Issue Type: Improvement
>    Affects Versions: 0.7.0
>            Reporter: Jason Lowe
>
> Currently AM credentials are explicitly kept separate from DAG credentials, but this can cause problems when credentials are automatically added to the application as part of the submission process but outside of the client's knowledge.  We need the ability for the AM's credentials to be merged with the DAG credentials so DAGs can pick up important credentials that were not submitted by the client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)