[jira] [Commented] (YARN-8071) Add ability to specify nodemanager environment variables individually

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16477863#comment-16477863 ] 

Jason Lowe commented on YARN-8071:
----------------------------------

Thanks for updating the patch!

bq. after further consideration, I put up a new patch that preserves the current behavior.  Not worth risking a compatibility issue for this.

Yeah, I agree.  I tried following the discussion in MAPREDUCE-3068 where it appears the behavior originated, and it's confusing.  It sounded like at one point the proposal was to override instead of append, but then it somehow shifted into only appending.  Anyway it's been this way since the beginning of YARN, and this JIRA is really only about the ability to specify environment variable settings as separate properties, not about changing the semantics of how values are combined with pre-existing values.  That can be tackled in a separate JIRA.

Patch looks good overall.  Curious, why was testPrependDistcache modified to no longer pass an empty set?  The modifications don't appear to be relevant or necessary.


> Add ability to specify nodemanager environment variables individually
> ---------------------------------------------------------------------
>
>                 Key: YARN-8071
>                 URL: https://issues.apache.org/jira/browse/YARN-8071
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: yarn
>    Affects Versions: 3.0.0
>            Reporter: Jim Brennan
>            Assignee: Jim Brennan
>            Priority: Major
>         Attachments: YARN-8071.001.patch, YARN-8071.002.patch, YARN-8071.003.patch
>
>
> YARN-6830 describes a problem where environment variables that contain commas cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>  
> -Dmapreduce.map.env.MODE=bar
>  -Dmapreduce.map.env.IMAGE_NAME=foo
>  -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what is actually being defined.
> {quote}
> The mapreduce properties were dealt with in [MAPREDUCE-7069].  This Jira will address the YARN properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org