[jira] [Commented] (YARN-2369) Environment variable handling assumes values should be appended

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14077729#comment-14077729 ] 

Jason Lowe commented on YARN-2369:
----------------------------------

The code in question is in org.apache.hadoop.yarn.util.Apps#addToEnvironment:

{code}
  public static void addToEnvironment(
      Map<String, String> environment,
      String variable, String value, String classPathSeparator) {
    String val = environment.get(variable);
    if (val == null) {
      val = value;
    } else {
      val = val + classPathSeparator + value;
    }
    environment.put(StringInterner.weakIntern(variable), 
        StringInterner.weakIntern(val));
  }
{code}

This has very surprising results for any variable that isn't path-like.  For example, we ran across a MapReduce job that had something like this in its environment settings:

yarn.app.mapreduce.am.env='JAVA_HOME=/inst/jdk,JAVA_HOME=/inst/jdk'

Rather than ending up with JAVA_HOME=/inst/jdk as one would expect, JAVA_HOME instead was set to /inst/jdk:/inst/jdk which completely broke the job.

It seems to me that we should either use a whitelist of variables that support appending or never append settings.  For the latter case if users desire values to be appended then they can ask for it explicitly in their variable settings, like one of these forms depending upon whether they want client-side environment variable expansion or container-side environment variable expansion:
{noformat}
PATH='$PATH:/my/extra/path'
PATH='{{PATH}}:/my/extra/path'
{noformat}

> Environment variable handling assumes values should be appended
> ---------------------------------------------------------------
>
>                 Key: YARN-2369
>                 URL: https://issues.apache.org/jira/browse/YARN-2369
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.2.0
>            Reporter: Jason Lowe
>
> When processing environment variables for a container context the code assumes that the value should be appended to any pre-existing value in the environment.  This may be desired behavior for handling path-like environment variables such as PATH, LD_LIBRARY_PATH, CLASSPATH, etc. but it is a non-intuitive and harmful way to handle any variable that does not have path-like semantics.



--
This message was sent by Atlassian JIRA
(v6.2#6252)