[jira] [Updated] (YARN-4727) Unable to override the $HADOOP_CONF_DIR env variable for container

["Jason Lowe (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jason Lowe updated YARN-4727:
-----------------------------
    Attachment: YARN-4727.001.patch

Attaching a patch that changes the {{putEnvIfNotNull}} to {{putEnvIfAbsent}} and added a unit test verifying the user can override HADOOP_CONF_DIR if desired.

> Unable to override the $HADOOP_CONF_DIR env variable for container
> ------------------------------------------------------------------
>
>                 Key: YARN-4727
>                 URL: https://issues.apache.org/jira/browse/YARN-4727
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.4.1, 2.5.2, 2.7.2, 2.6.4, 2.8.1
>            Reporter: Terence Yim
>            Assignee: Jason Lowe
>         Attachments: YARN-4727.001.patch
>
>
> Given the default config of "yarn.nodemanager.env-whitelist", application should be able to set the env variable $HADOOP_CONF_DIR to value other than the one in the NodeManager system environment. However, I believe due to a bug in the {{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch}} class, it is not possible so.
> From the {{sanitizeEnv()}} method in the ContainerLaunch class (https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java#L977)
> {noformat}
> putEnvIfNotNull(environment, 
>     Environment.HADOOP_CONF_DIR.name(), 
>     System.getenv(Environment.HADOOP_CONF_DIR.name())
>     );
> if (!Shell.WINDOWS) {
>   environment.put("JVM_PID", "$$");
> }
> String[] whitelist = conf.get(YarnConfiguration.NM_ENV_WHITELIST, YarnConfiguration.DEFAULT_NM_ENV_WHITELIST).split(",");
>     
> for(String whitelistEnvVariable : whitelist) {
>   putEnvIfAbsent(environment, whitelistEnvVariable.trim());
> }
> ...
> private static void putEnvIfAbsent(
>     Map<String, String> environment, String variable) {
>   if (environment.get(variable) == null) {
>     putEnvIfNotNull(environment, variable, System.getenv(variable));
>   }
> }
> {noformat}
> So there two issues here.
> 1. the environment is already set with the system environment of the NM in the {{putEnvIfNotNull}} call, hence the {{putEnvIfAbsent}} call will never set it to some new value
> 2. Inside the {{putEnvIfAbsent}} call, it uses the system environment of the NM, which it should be using the one from the {{launchContext}} instead.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org