You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Prabhu Joseph (Jira)" <ji...@apache.org> on 2022/09/21 11:25:00 UTC

[jira] [Resolved] (YARN-11255) Support loading alternative docker client config from system environment

     [ https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Prabhu Joseph resolved YARN-11255.
----------------------------------
    Fix Version/s: 3.4.0
       Resolution: Fixed

> Support loading alternative docker client config from system environment
> ------------------------------------------------------------------------
>
>                 Key: YARN-11255
>                 URL: https://issues.apache.org/jira/browse/YARN-11255
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Ashutosh Gupta
>            Assignee: Ashutosh Gupta
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.4.0
>
>
> When using YARN docker support, although the hadoop shell supported 
> {code:java}
> -docker_client_config{code}
>  to pass the client config file that contains security token to generate the docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the docker setting via system environment e.g. 
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in spark code or others
> Eg
> When using remote container registry, the {{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
> file containing the credentials used to authenticate.
> {code:java}
> DOCKER_IMAGE_NAME=hadoop-docker 
> DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
> spark-submit --master yarn \
> --deploy-mode cluster \
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG \
> sparkR.R{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org