You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2018/02/07 16:11:00 UTC
[jira] [Comment Edited] (YARN-5714) ContainerExecutor does not
order environment map
[ https://issues.apache.org/jira/browse/YARN-5714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16350359#comment-16350359 ]
Jason Lowe edited comment on YARN-5714 at 2/7/18 4:10 PM:
----------------------------------------------------------
On second thought, it's extremely unlikely that NM whitelist variables could reference user variables. Details are in [this comment|https://issues.apache.org/jira/browse/YARN-7677?focusedCommentId=16350353&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16350353] on YARN-7677. I think we'll be fine if we make sure the NM whitelist inherited variables appear first in the launch script then followed by the user's variables in the order they are specified in the container launch context. YARN-7677 should be taking care of the NM whitelist variables, so this JIRA can tackle ordering the user's variables.
was (Author: jlowe):
On second thought, it's extremely unlikely that NM whitelist variables could reference user variables. Details are in [this comment|https://issues.apache.org/jira/browse/YARN-7677?focusedCommentId=16350353&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16350353] on YARN-7667. I think we'll be fine if we make sure the NM whitelist inherited variables appear first in the launch script then followed by the user's variables in the order they are specified in the container launch context. YARN-7667 should be taking care of the NM whitelist variables, so this JIRA can tackle ordering the user's variables.
> ContainerExecutor does not order environment map
> ------------------------------------------------
>
> Key: YARN-5714
> URL: https://issues.apache.org/jira/browse/YARN-5714
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.4.1, 2.5.2, 2.7.3, 2.6.4, 3.0.0-alpha1
> Environment: all (linux and windows alike)
> Reporter: Remi Catherinot
> Assignee: Remi Catherinot
> Priority: Trivial
> Labels: oct16-medium
> Attachments: YARN-5714.001.patch, YARN-5714.002.patch, YARN-5714.003.patch, YARN-5714.004.patch, YARN-5714.005.patch, YARN-5714.006.patch
>
> Original Estimate: 120h
> Remaining Estimate: 120h
>
> when dumping the launch container script, environment variables are dumped based on the order internally used by the map implementation (hash based). It does not take into consideration that some env varibales may refer each other, and so that some env variables must be declared before those referencing them.
> In my case, i ended up having LD_LIBRARY_PATH which was depending on HADOOP_COMMON_HOME being dumped before HADOOP_COMMON_HOME. Thus it had a wrong value and so native libraries weren't loaded. jobs were running but not at their best efficiency. This is just a use case falling into that bug, but i'm sure others may happen as well.
> I already have a patch running in my production environment, i just estimate to 5 days for packaging the patch in the right fashion for JIRA + try my best to add tests.
> Note : the patch is not OS aware with a default empty implementation. I will only implement the unix version on a 1st release. I'm not used to windows env variables syntax so it will take me more time/research for it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org