You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by sh...@inmobi.com on 2014/12/03 10:14:45 UTC
Review Request 28637: OOZIE-2030 Configuration properties from global
section is not getting set in Hadoop job conf when using sub-workflow
action in Oozie workflow.xml
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28637/
-----------------------------------------------------------
Review request for oozie.
Bugs: OOZIE-2030
https://issues.apache.org/jira/browse/OOZIE-2030
Repository: oozie-git
Description
-------
1. Propagate properties from configuration and job-xml from parent workflow to sub-workflow. Currently, its propagated as workflow properties, but they are required in action conf
2. For java actions, add properties from configuration and job-xml to launcher job
Diffs
-----
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java f207d74
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java f207d74
core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java bda34b5
core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java bda34b5
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 48166a5
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 48166a5
core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 0d7e926
core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 0d7e926
examples/src/main/apps/subwf/job.xml PRE-CREATION
examples/src/main/apps/subwf/workflow.xml 9873a50
examples/src/main/apps/subwf/workflow.xml 9873a50
Diff: https://reviews.apache.org/r/28637/diff/
Testing
-------
UTs and end to end test
Thanks,
shwethags
Re: Review Request 28637: OOZIE-2030 Configuration properties from
global
section is not getting set in Hadoop job conf when using sub-workflow
action in Oozie workflow.xml
Posted by sh...@inmobi.com.
> On Dec. 11, 2014, 9:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 242
> > <https://reviews.apache.org/r/28637/diff/3/?file=785337#file785337line242>
> >
> > I think it actually makes sense and whatever I thought might conflict would be ok. Could you still add a config oozie.launcher.only.job.copy.actionconf (or some better name) as bother server and action level config with default value as true, so that we can turn it off in production if some unexpected conflict rises. If only few users encounter issue we can ask them to override at action level and if there is a major issue can turn it off at server level itself.
Will do
> On Dec. 11, 2014, 9:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java, line 180
> > <https://reviews.apache.org/r/28637/diff/3/?file=785338#file785338line180>
> >
> > Based on the code in LiteWorkflowAppParser.handleGlobal, eConf.getChild("configuration", ns) should contain the global configuration. Instead of parsing global and jobxml again, you could just use that.
> >
> > I think the cleaner way to do this would be to not touch ActionExecutors at all. Should parse and store global configuration in WorkflowInstance once and add it to each action configuration and also initialize global configuration of sub WorkflowInstance created with it. Currently the global section and its job.xml is parsed for each action which is really bad for latency. This would also take care of propagating it to nested sub-workflows. If the sub-workflow had global configuration then that should be merged while parsing itself to the parent global config initialized.
Makes sense. But that change will be more involved. Lets track that in another jira
- shwethags
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28637/#review64788
-----------------------------------------------------------
On Dec. 8, 2014, 7:20 a.m., shwethags wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28637/
> -----------------------------------------------------------
>
> (Updated Dec. 8, 2014, 7:20 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-2030
> https://issues.apache.org/jira/browse/OOZIE-2030
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> 1. Propagate properties from configuration and job-xml from parent workflow to sub-workflow. Currently, its propagated as workflow properties, but they are required in action conf
> 2. For java actions, add properties from configuration and job-xml to launcher job
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/action/ActionExecutor.java ff836fb
> core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java fed1d7a
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java f207d74
> core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java bda34b5
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 18de48a
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 48166a5
> core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 0d7e926
> examples/src/main/apps/subwf/job.xml PRE-CREATION
> examples/src/main/apps/subwf/workflow.xml 9873a50
>
> Diff: https://reviews.apache.org/r/28637/diff/
>
>
> Testing
> -------
>
> UTs and end to end test
>
>
> Thanks,
>
> shwethags
>
>
Re: Review Request 28637: OOZIE-2030 Configuration properties from
global
section is not getting set in Hadoop job conf when using sub-workflow
action in Oozie workflow.xml
Posted by Rohini Palaniswamy <ro...@gmail.com>.
> On Dec. 11, 2014, 9:21 p.m., Rohini Palaniswamy wrote:
> > core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java, line 180
> > <https://reviews.apache.org/r/28637/diff/3/?file=785338#file785338line180>
> >
> > Based on the code in LiteWorkflowAppParser.handleGlobal, eConf.getChild("configuration", ns) should contain the global configuration. Instead of parsing global and jobxml again, you could just use that.
> >
> > I think the cleaner way to do this would be to not touch ActionExecutors at all. Should parse and store global configuration in WorkflowInstance once and add it to each action configuration and also initialize global configuration of sub WorkflowInstance created with it. Currently the global section and its job.xml is parsed for each action which is really bad for latency. This would also take care of propagating it to nested sub-workflows. If the sub-workflow had global configuration then that should be merged while parsing itself to the parent global config initialized.
>
> shwethags wrote:
> Makes sense. But that change will be more involved. Lets track that in another jira
Just updating comment to the discussion I had with Shwetha. This will be the cleaner way of fixing this jira and cannot be done in another jira. Logically and with the current separation of work, it is not clean and crossing boundaries to parse workflow and job xml files for global section in ActionExecutors. They should be done during the actual parsing and construction of actions. ActionExecutors should only be for executing the action.
- Rohini
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28637/#review64788
-----------------------------------------------------------
On Dec. 8, 2014, 7:20 a.m., shwethags wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28637/
> -----------------------------------------------------------
>
> (Updated Dec. 8, 2014, 7:20 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-2030
> https://issues.apache.org/jira/browse/OOZIE-2030
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> 1. Propagate properties from configuration and job-xml from parent workflow to sub-workflow. Currently, its propagated as workflow properties, but they are required in action conf
> 2. For java actions, add properties from configuration and job-xml to launcher job
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/action/ActionExecutor.java ff836fb
> core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java fed1d7a
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java f207d74
> core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java bda34b5
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 18de48a
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 48166a5
> core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 0d7e926
> examples/src/main/apps/subwf/job.xml PRE-CREATION
> examples/src/main/apps/subwf/workflow.xml 9873a50
>
> Diff: https://reviews.apache.org/r/28637/diff/
>
>
> Testing
> -------
>
> UTs and end to end test
>
>
> Thanks,
>
> shwethags
>
>
Re: Review Request 28637: OOZIE-2030 Configuration properties from
global
section is not getting set in Hadoop job conf when using sub-workflow
action in Oozie workflow.xml
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28637/#review64788
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/28637/#comment107555>
I think it actually makes sense and whatever I thought might conflict would be ok. Could you still add a config oozie.launcher.only.job.copy.actionconf (or some better name) as bother server and action level config with default value as true, so that we can turn it off in production if some unexpected conflict rises. If only few users encounter issue we can ask them to override at action level and if there is a major issue can turn it off at server level itself.
core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java
<https://reviews.apache.org/r/28637/#comment107556>
Based on the code in LiteWorkflowAppParser.handleGlobal, eConf.getChild("configuration", ns) should contain the global configuration. Instead of parsing global and jobxml again, you could just use that.
I think the cleaner way to do this would be to not touch ActionExecutors at all. Should parse and store global configuration in WorkflowInstance once and add it to each action configuration and also initialize global configuration of sub WorkflowInstance created with it. Currently the global section and its job.xml is parsed for each action which is really bad for latency. This would also take care of propagating it to nested sub-workflows. If the sub-workflow had global configuration then that should be merged while parsing itself to the parent global config initialized.
- Rohini Palaniswamy
On Dec. 8, 2014, 7:20 a.m., shwethags wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28637/
> -----------------------------------------------------------
>
> (Updated Dec. 8, 2014, 7:20 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-2030
> https://issues.apache.org/jira/browse/OOZIE-2030
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> 1. Propagate properties from configuration and job-xml from parent workflow to sub-workflow. Currently, its propagated as workflow properties, but they are required in action conf
> 2. For java actions, add properties from configuration and job-xml to launcher job
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/action/ActionExecutor.java ff836fb
> core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java fed1d7a
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java f207d74
> core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java bda34b5
> core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 18de48a
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 48166a5
> core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 0d7e926
> examples/src/main/apps/subwf/job.xml PRE-CREATION
> examples/src/main/apps/subwf/workflow.xml 9873a50
>
> Diff: https://reviews.apache.org/r/28637/diff/
>
>
> Testing
> -------
>
> UTs and end to end test
>
>
> Thanks,
>
> shwethags
>
>
Re: Review Request 28637: OOZIE-2030 Configuration properties from
global
section is not getting set in Hadoop job conf when using sub-workflow
action in Oozie workflow.xml
Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28637/
-----------------------------------------------------------
(Updated Dec. 8, 2014, 7:20 a.m.)
Review request for oozie.
Changes
-------
Moved parseJobXmlAndConfiguration() from JavaActionExecutor to ActionExecutor
Bugs: OOZIE-2030
https://issues.apache.org/jira/browse/OOZIE-2030
Repository: oozie-git
Description
-------
1. Propagate properties from configuration and job-xml from parent workflow to sub-workflow. Currently, its propagated as workflow properties, but they are required in action conf
2. For java actions, add properties from configuration and job-xml to launcher job
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/action/ActionExecutor.java ff836fb
core/src/main/java/org/apache/oozie/action/hadoop/FsActionExecutor.java fed1d7a
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java f207d74
core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java bda34b5
core/src/main/java/org/apache/oozie/service/HadoopAccessorService.java 18de48a
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 48166a5
core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 0d7e926
examples/src/main/apps/subwf/job.xml PRE-CREATION
examples/src/main/apps/subwf/workflow.xml 9873a50
Diff: https://reviews.apache.org/r/28637/diff/
Testing
-------
UTs and end to end test
Thanks,
shwethags
Re: Review Request 28637: OOZIE-2030 Configuration properties from
global
section is not getting set in Hadoop job conf when using sub-workflow
action in Oozie workflow.xml
Posted by Jaydeep Vishwakarma <ja...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28637/#review63970
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java
<https://reviews.apache.org/r/28637/#comment106345>
this method looks like common can be move it to some common place, it has nothing specific to do with JavaactionExecutor.
- Jaydeep Vishwakarma
On Dec. 3, 2014, 9:20 a.m., shwethags wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28637/
> -----------------------------------------------------------
>
> (Updated Dec. 3, 2014, 9:20 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-2030
> https://issues.apache.org/jira/browse/OOZIE-2030
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> 1. Propagate properties from configuration and job-xml from parent workflow to sub-workflow. Currently, its propagated as workflow properties, but they are required in action conf
> 2. For java actions, add properties from configuration and job-xml to launcher job
>
>
> Diffs
> -----
>
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java f207d74
> core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java bda34b5
> core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 48166a5
> core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 0d7e926
> examples/src/main/apps/subwf/job.xml PRE-CREATION
> examples/src/main/apps/subwf/workflow.xml 9873a50
>
> Diff: https://reviews.apache.org/r/28637/diff/
>
>
> Testing
> -------
>
> UTs and end to end test
>
>
> Thanks,
>
> shwethags
>
>
Re: Review Request 28637: OOZIE-2030 Configuration properties from
global
section is not getting set in Hadoop job conf when using sub-workflow
action in Oozie workflow.xml
Posted by sh...@inmobi.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28637/
-----------------------------------------------------------
(Updated Dec. 3, 2014, 9:20 a.m.)
Review request for oozie.
Bugs: OOZIE-2030
https://issues.apache.org/jira/browse/OOZIE-2030
Repository: oozie-git
Description
-------
1. Propagate properties from configuration and job-xml from parent workflow to sub-workflow. Currently, its propagated as workflow properties, but they are required in action conf
2. For java actions, add properties from configuration and job-xml to launcher job
Diffs (updated)
-----
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java f207d74
core/src/main/java/org/apache/oozie/action/oozie/SubWorkflowActionExecutor.java bda34b5
core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 48166a5
core/src/test/java/org/apache/oozie/action/oozie/TestSubWorkflowActionExecutor.java 0d7e926
examples/src/main/apps/subwf/job.xml PRE-CREATION
examples/src/main/apps/subwf/workflow.xml 9873a50
Diff: https://reviews.apache.org/r/28637/diff/
Testing
-------
UTs and end to end test
Thanks,
shwethags