[jira] [Commented] (OOZIE-2896) Ensure compatibility for existing LauncherMapper settings

["Andras Piros (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16188186#comment-16188186 ] 

Andras Piros commented on OOZIE-2896:
-------------------------------------

[~pbacsko] here is how things are rolling regarding expected code changes:
# extract [*{{JavaActionExecutor#injectLauncherProperties()}}*|https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L305] so that it is more testable / extendable
# maintain a white list of MapReduce v1, MapReduce V2, and YARN configuration values configured with {{oozie.launcher.}} prefix, and put without the prefix to launcher {{Configuration}}:
** {{mapred.map.max.attempts}} / {{mapreduce.map.maxattempts}}
** {{mapred.job.map.memory.mb}} / {{mapreduce.map.memory.mb}}
** {{mapreduce.map.cpu.vcores}}
** {{mapred.map.child.log.level}} / {{mapreduce.map.log.level}}
** {{mapred.child.java.opts}}
** {{mapred.map.child.java.opts}} / {{mapreduce.map.java.opts}}
** {{mapred.child.env}}
** {{mapred.map.child.env}} / {{mapreduce.map.env}}
** {{mapred.job.priority}} / {{mapreduce.job.priority}}
** {{mapred.job.queue.name}} / {{mapreduce.job.queuename}}
** {{yarn.app.mapreduce.am.env}}
** {{yarn.app.mapreduce.am.admin.user.env}}
** {{yarn.app.mapreduce.am.command-opts}}
** {{yarn.app.mapreduce.am.admin-command-opts}}
** {{yarn.app.mapreduce.am.resource.mb}}
** {{yarn.app.mapreduce.am.resource.cpu-vcores}}
# when some action property is not on the whitelist, emit a {{WARN}} log that this property won't be taken care of; it's outside of backwards compatibility guarantees. However, still attach to the launcher {{Configuration}} without a prefix
# for properties within the same group, where the user has defined at least two different properties, emit an {{ERROR}} log stating that the user has inserted contradictory parameter values, and die. Such groups are:
** environment variables:
*** MapReduce v1: {{mapred.map.env}} / {{mapred.child.env}} / {{mapred.map.child.env}}
*** MapReduce v2: {{mapreduce.map.env}}
*** YARN: {{yarn.app.mapreduce.am.env}} / {{yarn.app.mapreduce.am.admin.user.env}}
** JVM options:
*** MapReduce v1: {{mapred.child.java.opts}} / {{mapred.map.child.java.opts}}
*** MapReduce v2: {{mapreduce.child.java.opts}}
*** YARN: {{yarn.app.mapreduce.am.command-opts}} / {{yarn.app.mapreduce.am.admin-command-opts}}

[~rkanter] [~rohini] [~pbacsko] [~gezapeti] please express your thoughts.

> Ensure compatibility for existing LauncherMapper settings
> ---------------------------------------------------------
>
>                 Key: OOZIE-2896
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2896
>             Project: Oozie
>          Issue Type: Sub-task
>    Affects Versions: 5.0.0
>            Reporter: Peter Bacsko
>            Assignee: Andras Piros
>            Priority: Blocker
>
> There are a couple of LauncherMapper settings that we should support for backward compatibility reasons.
> For example, {{oozie.launcher.mapreduce.map.memory.mb}}, {{mapred.child.env}}, {{mapred.map.env}}, etc.
> We also have to discuss the list of properties we're going to support, since there are many of them.
> List of hadoop properties under this task:
> mapreduce.job.queuename
> mapreduce.map.memory.mb
> mapreduce.map.java.opts
> mapreduce.map.env
> mapreduce.job.priority
> yarn.app.mapreduce.am.scheduler.heartbeat.interval-ms
> yarn.app.mapreduce.am.resource.mb
> yarn.app.mapreduce.am.command-opts
> yarn.app.mapreduce.am.env
> ACL related properties should go under OOZIE-2897:
> mapreduce.job.acl-view-job
> mapreduce.job.acl-modfy-job



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)