You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "lynn (Jira)" <ji...@apache.org> on 2019/10/16 06:17:00 UTC

[jira] [Updated] (YARN-9906) When setting multi volumes throurh the "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS" setting is not valid

     [ https://issues.apache.org/jira/browse/YARN-9906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

lynn updated YARN-9906:
-----------------------
    Description: 
As [https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/DockerContainers.html#Application_Submission] described, when I set the item "{{YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS" to multi volumes mounts, the value is a comma-separated list of mounts.}}
 vars="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker, YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/etc/passwd:/etc/passwd:ro,/etc/group:/etc/group:ro;/etc/hadoop/conf:/etc/hadoop/conf"
 hadoop jar hadoop-examples.jar pi -Dyarn.app.mapreduce.am.env=$vars \
 -Dmapreduce.map.env=$vars -Dmapreduce.reduce.env=$vars 10 100

I found the docker container can mount the first volume, so it can't be running successfully without report error!

The code of [DockerLinuxContainerRuntime.java|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java] as follows:

if (environment.containsKey(ENV_DOCKER_CONTAINER_MOUNTS)) {
      Matcher parsedMounts = USER_MOUNT_PATTERN.matcher(
          environment.get(ENV_DOCKER_CONTAINER_MOUNTS));
      if (!parsedMounts.find()) {
        throw new ContainerExecutionException(
            "Unable to parse user supplied mount list: "
                + environment.get(ENV_DOCKER_CONTAINER_MOUNTS));
      }

The regex pattern is in [OCIContainerRuntime|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/OCIContainerRuntime.java] as follows

static final Pattern USER_MOUNT_PATTERN = Pattern.compile(
      "(?<=^|,)([^:\\x00]+):([^:\\x00]+)" +
      "(:(r[ow]|(r[ow][+])?(r?shared|r?slave|r?private)))?(?:,|$)");

it is seperated by comma indeed, but when i read the code of submit the jar to yarn , i find the code [Apps.java|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java]
private static final Pattern VARVAL_SPLITTER = Pattern.compile(
        "(?<=^|,)"                            // preceded by ',' or line begin
      + '(' + Shell.ENV_NAME_REGEX + ')'      // var group
      + '='
      + "([^,]*)"                             // val group
      );

It is sepearted by comma as the same.

So, I just modify the comma to semicolon(";") of the item "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS".



  was:
As [https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/DockerContainers.html#Application_Submission] described, when I set the item "{{YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS" to multi volumes mounts, the value is a comma-separated list of mounts.}}
vars="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker, YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/etc/passwd:/etc/passwd:ro,/etc/group:/etc/group:ro;/etc/hadoop/conf:/etc/hadoop/conf"
hadoop jar hadoop-examples.jar pi -Dyarn.app.mapreduce.am.env=$vars \
        -Dmapreduce.map.env=$vars -Dmapreduce.reduce.env=$vars 10 100


I found the docker container can mount the first volume, so it can't be running successfully without report error!


> When setting multi volumes throurh the "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS" setting is not  valid
> ---------------------------------------------------------------------------------------------------
>
>                 Key: YARN-9906
>                 URL: https://issues.apache.org/jira/browse/YARN-9906
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>            Reporter: lynn
>            Priority: Major
>
> As [https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/DockerContainers.html#Application_Submission] described, when I set the item "{{YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS" to multi volumes mounts, the value is a comma-separated list of mounts.}}
>  vars="YARN_CONTAINER_RUNTIME_TYPE=docker,YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker, YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=/etc/passwd:/etc/passwd:ro,/etc/group:/etc/group:ro;/etc/hadoop/conf:/etc/hadoop/conf"
>  hadoop jar hadoop-examples.jar pi -Dyarn.app.mapreduce.am.env=$vars \
>  -Dmapreduce.map.env=$vars -Dmapreduce.reduce.env=$vars 10 100
> I found the docker container can mount the first volume, so it can't be running successfully without report error!
> The code of [DockerLinuxContainerRuntime.java|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java] as follows:
> if (environment.containsKey(ENV_DOCKER_CONTAINER_MOUNTS)) {
>       Matcher parsedMounts = USER_MOUNT_PATTERN.matcher(
>           environment.get(ENV_DOCKER_CONTAINER_MOUNTS));
>       if (!parsedMounts.find()) {
>         throw new ContainerExecutionException(
>             "Unable to parse user supplied mount list: "
>                 + environment.get(ENV_DOCKER_CONTAINER_MOUNTS));
>       }
> The regex pattern is in [OCIContainerRuntime|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/OCIContainerRuntime.java] as follows
> static final Pattern USER_MOUNT_PATTERN = Pattern.compile(
>       "(?<=^|,)([^:\\x00]+):([^:\\x00]+)" +
>       "(:(r[ow]|(r[ow][+])?(r?shared|r?slave|r?private)))?(?:,|$)");
> it is seperated by comma indeed, but when i read the code of submit the jar to yarn , i find the code [Apps.java|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java]
> private static final Pattern VARVAL_SPLITTER = Pattern.compile(
>         "(?<=^|,)"                            // preceded by ',' or line begin
>       + '(' + Shell.ENV_NAME_REGEX + ')'      // var group
>       + '='
>       + "([^,]*)"                             // val group
>       );
> It is sepearted by comma as the same.
> So, I just modify the comma to semicolon(";") of the item "YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org