You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by sh...@inmobi.com on 2014/09/09 11:24:16 UTC
Re: Review Request 24165: OOZIE-1954: Add a way for the MapReduce
action to be configured by Java code
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24165/#review52696
-----------------------------------------------------------
docs/src/site/twiki/WorkflowFunctionalSpec.twiki
<https://reviews.apache.org/r/24165/#comment91676>
Instead of depending on oozie artifact for the interface(which is difficult to get given that oozie artifacts are not available in any mvn repo), can the interface be org.apache.hadoop.util.Tool. The user provided run() can update the Conf.
run() doesn't sound like setup, but serves the purpose with conf.
- shwethags
On July 31, 2014, 10:33 p.m., Robert Kanter wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24165/
> -----------------------------------------------------------
>
> (Updated July 31, 2014, 10:33 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1954
> https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Diffs
> -----
>
> client/src/main/resources/oozie-workflow-0.5.xsd 6620a4e
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 94b55cf
> core/src/test/java/org/apache/oozie/action/hadoop/MapperReducerForTest.java 7fe1f42
> docs/src/site/twiki/WorkflowFunctionalSpec.twiki 3319bcc
> examples/src/main/apps/map-reduce/job-with-config-class.properties PRE-CREATION
> examples/src/main/apps/map-reduce/job.properties 7b7a24c
> examples/src/main/apps/map-reduce/workflow-with-config-class.xml PRE-CREATION
> examples/src/main/java/org/apache/oozie/example/SampleOozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 2cb690f
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 322f498
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java f417a78
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorException.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 67a2a43
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorForTest.java PRE-CREATION
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java e78c98a
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java bcc22f2
>
> Diff: https://reviews.apache.org/r/24165/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Robert Kanter
>
>
Re: Review Request 24165: OOZIE-1954: Add a way for the MapReduce
action to be configured by Java code
Posted by Robert Kanter <rk...@cloudera.com>.
> On Sept. 9, 2014, 9:24 a.m., shwethags wrote:
> > docs/src/site/twiki/WorkflowFunctionalSpec.twiki, line 605
> > <https://reviews.apache.org/r/24165/diff/1/?file=647181#file647181line605>
> >
> > Instead of depending on oozie artifact for the interface(which is difficult to get given that oozie artifacts are not available in any mvn repo), can the interface be org.apache.hadoop.util.Tool. The user provided run() can update the Conf.
> >
> > run() doesn't sound like setup, but serves the purpose with conf.
>
> Robert Kanter wrote:
> Using Tool gives us less control if we want to make any changes (e.g. add new methods, etc) in the future, so I'd prefer to use an Oozie-specific interface. It's also more clear what this is for.
>
> I think the real solution for that difficulting is to publish artifacts, which many users have been asking for, and everyone seems to agree that we should do. I'm hoping that we'll publish starting with the 4.1 release.
I've talked with Bowen offline and he's going to publish the 4.1 (OOZIE-1842). And by the time we actually release OOZIE-1954, publishing to maven should be part of the release process.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24165/#review52696
-----------------------------------------------------------
On July 31, 2014, 10:33 p.m., Robert Kanter wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24165/
> -----------------------------------------------------------
>
> (Updated July 31, 2014, 10:33 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1954
> https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Diffs
> -----
>
> client/src/main/resources/oozie-workflow-0.5.xsd 6620a4e
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 94b55cf
> core/src/test/java/org/apache/oozie/action/hadoop/MapperReducerForTest.java 7fe1f42
> docs/src/site/twiki/WorkflowFunctionalSpec.twiki 3319bcc
> examples/src/main/apps/map-reduce/job-with-config-class.properties PRE-CREATION
> examples/src/main/apps/map-reduce/job.properties 7b7a24c
> examples/src/main/apps/map-reduce/workflow-with-config-class.xml PRE-CREATION
> examples/src/main/java/org/apache/oozie/example/SampleOozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 2cb690f
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 322f498
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java f417a78
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorException.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 67a2a43
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorForTest.java PRE-CREATION
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java e78c98a
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java bcc22f2
>
> Diff: https://reviews.apache.org/r/24165/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Robert Kanter
>
>
Re: Review Request 24165: OOZIE-1954: Add a way for the MapReduce
action to be configured by Java code
Posted by sh...@inmobi.com.
> On Sept. 9, 2014, 9:24 a.m., shwethags wrote:
> > docs/src/site/twiki/WorkflowFunctionalSpec.twiki, line 605
> > <https://reviews.apache.org/r/24165/diff/1/?file=647181#file647181line605>
> >
> > Instead of depending on oozie artifact for the interface(which is difficult to get given that oozie artifacts are not available in any mvn repo), can the interface be org.apache.hadoop.util.Tool. The user provided run() can update the Conf.
> >
> > run() doesn't sound like setup, but serves the purpose with conf.
>
> Robert Kanter wrote:
> Using Tool gives us less control if we want to make any changes (e.g. add new methods, etc) in the future, so I'd prefer to use an Oozie-specific interface. It's also more clear what this is for.
>
> I think the real solution for that difficulting is to publish artifacts, which many users have been asking for, and everyone seems to agree that we should do. I'm hoping that we'll publish starting with the 4.1 release.
>
> Robert Kanter wrote:
> I've talked with Bowen offline and he's going to publish the 4.1 (OOZIE-1842). And by the time we actually release OOZIE-1954, publishing to maven should be part of the release process.
>
> shwethags wrote:
> I am fine with it if the oozie artifacts are available.
>
> If we move the interface to oozie-client, then the launcher job will need oozie-client jar as well?
>
> Robert Kanter wrote:
> Correct. I think putting it in the oozie-sharelib (or perhaps a new artifact?) makes the most sense.
If all the oozie artifacts are in mvn repo, we can go with oozie-sharelib itself
- shwethags
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24165/#review52696
-----------------------------------------------------------
On July 31, 2014, 10:33 p.m., Robert Kanter wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24165/
> -----------------------------------------------------------
>
> (Updated July 31, 2014, 10:33 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1954
> https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Diffs
> -----
>
> client/src/main/resources/oozie-workflow-0.5.xsd 6620a4e
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 94b55cf
> core/src/test/java/org/apache/oozie/action/hadoop/MapperReducerForTest.java 7fe1f42
> docs/src/site/twiki/WorkflowFunctionalSpec.twiki 3319bcc
> examples/src/main/apps/map-reduce/job-with-config-class.properties PRE-CREATION
> examples/src/main/apps/map-reduce/job.properties 7b7a24c
> examples/src/main/apps/map-reduce/workflow-with-config-class.xml PRE-CREATION
> examples/src/main/java/org/apache/oozie/example/SampleOozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 2cb690f
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 322f498
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java f417a78
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorException.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 67a2a43
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorForTest.java PRE-CREATION
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java e78c98a
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java bcc22f2
>
> Diff: https://reviews.apache.org/r/24165/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Robert Kanter
>
>
Re: Review Request 24165: OOZIE-1954: Add a way for the MapReduce
action to be configured by Java code
Posted by Robert Kanter <rk...@cloudera.com>.
> On Sept. 9, 2014, 9:24 a.m., shwethags wrote:
> > docs/src/site/twiki/WorkflowFunctionalSpec.twiki, line 605
> > <https://reviews.apache.org/r/24165/diff/1/?file=647181#file647181line605>
> >
> > Instead of depending on oozie artifact for the interface(which is difficult to get given that oozie artifacts are not available in any mvn repo), can the interface be org.apache.hadoop.util.Tool. The user provided run() can update the Conf.
> >
> > run() doesn't sound like setup, but serves the purpose with conf.
>
> Robert Kanter wrote:
> Using Tool gives us less control if we want to make any changes (e.g. add new methods, etc) in the future, so I'd prefer to use an Oozie-specific interface. It's also more clear what this is for.
>
> I think the real solution for that difficulting is to publish artifacts, which many users have been asking for, and everyone seems to agree that we should do. I'm hoping that we'll publish starting with the 4.1 release.
>
> Robert Kanter wrote:
> I've talked with Bowen offline and he's going to publish the 4.1 (OOZIE-1842). And by the time we actually release OOZIE-1954, publishing to maven should be part of the release process.
>
> shwethags wrote:
> I am fine with it if the oozie artifacts are available.
>
> If we move the interface to oozie-client, then the launcher job will need oozie-client jar as well?
Correct. I think putting it in the oozie-sharelib (or perhaps a new artifact?) makes the most sense.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24165/#review52696
-----------------------------------------------------------
On July 31, 2014, 10:33 p.m., Robert Kanter wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24165/
> -----------------------------------------------------------
>
> (Updated July 31, 2014, 10:33 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1954
> https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Diffs
> -----
>
> client/src/main/resources/oozie-workflow-0.5.xsd 6620a4e
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 94b55cf
> core/src/test/java/org/apache/oozie/action/hadoop/MapperReducerForTest.java 7fe1f42
> docs/src/site/twiki/WorkflowFunctionalSpec.twiki 3319bcc
> examples/src/main/apps/map-reduce/job-with-config-class.properties PRE-CREATION
> examples/src/main/apps/map-reduce/job.properties 7b7a24c
> examples/src/main/apps/map-reduce/workflow-with-config-class.xml PRE-CREATION
> examples/src/main/java/org/apache/oozie/example/SampleOozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 2cb690f
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 322f498
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java f417a78
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorException.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 67a2a43
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorForTest.java PRE-CREATION
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java e78c98a
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java bcc22f2
>
> Diff: https://reviews.apache.org/r/24165/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Robert Kanter
>
>
Re: Review Request 24165: OOZIE-1954: Add a way for the MapReduce
action to be configured by Java code
Posted by sh...@inmobi.com.
> On Sept. 9, 2014, 9:24 a.m., shwethags wrote:
> > docs/src/site/twiki/WorkflowFunctionalSpec.twiki, line 605
> > <https://reviews.apache.org/r/24165/diff/1/?file=647181#file647181line605>
> >
> > Instead of depending on oozie artifact for the interface(which is difficult to get given that oozie artifacts are not available in any mvn repo), can the interface be org.apache.hadoop.util.Tool. The user provided run() can update the Conf.
> >
> > run() doesn't sound like setup, but serves the purpose with conf.
>
> Robert Kanter wrote:
> Using Tool gives us less control if we want to make any changes (e.g. add new methods, etc) in the future, so I'd prefer to use an Oozie-specific interface. It's also more clear what this is for.
>
> I think the real solution for that difficulting is to publish artifacts, which many users have been asking for, and everyone seems to agree that we should do. I'm hoping that we'll publish starting with the 4.1 release.
>
> Robert Kanter wrote:
> I've talked with Bowen offline and he's going to publish the 4.1 (OOZIE-1842). And by the time we actually release OOZIE-1954, publishing to maven should be part of the release process.
I am fine with it if the oozie artifacts are available.
If we move the interface to oozie-client, then the launcher job will need oozie-client jar as well?
- shwethags
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24165/#review52696
-----------------------------------------------------------
On July 31, 2014, 10:33 p.m., Robert Kanter wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24165/
> -----------------------------------------------------------
>
> (Updated July 31, 2014, 10:33 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1954
> https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Diffs
> -----
>
> client/src/main/resources/oozie-workflow-0.5.xsd 6620a4e
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 94b55cf
> core/src/test/java/org/apache/oozie/action/hadoop/MapperReducerForTest.java 7fe1f42
> docs/src/site/twiki/WorkflowFunctionalSpec.twiki 3319bcc
> examples/src/main/apps/map-reduce/job-with-config-class.properties PRE-CREATION
> examples/src/main/apps/map-reduce/job.properties 7b7a24c
> examples/src/main/apps/map-reduce/workflow-with-config-class.xml PRE-CREATION
> examples/src/main/java/org/apache/oozie/example/SampleOozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 2cb690f
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 322f498
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java f417a78
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorException.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 67a2a43
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorForTest.java PRE-CREATION
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java e78c98a
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java bcc22f2
>
> Diff: https://reviews.apache.org/r/24165/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Robert Kanter
>
>
Re: Review Request 24165: OOZIE-1954: Add a way for the MapReduce
action to be configured by Java code
Posted by Robert Kanter <rk...@cloudera.com>.
> On Sept. 9, 2014, 9:24 a.m., shwethags wrote:
> > docs/src/site/twiki/WorkflowFunctionalSpec.twiki, line 605
> > <https://reviews.apache.org/r/24165/diff/1/?file=647181#file647181line605>
> >
> > Instead of depending on oozie artifact for the interface(which is difficult to get given that oozie artifacts are not available in any mvn repo), can the interface be org.apache.hadoop.util.Tool. The user provided run() can update the Conf.
> >
> > run() doesn't sound like setup, but serves the purpose with conf.
Using Tool gives us less control if we want to make any changes (e.g. add new methods, etc) in the future, so I'd prefer to use an Oozie-specific interface. It's also more clear what this is for.
I think the real solution for that difficulting is to publish artifacts, which many users have been asking for, and everyone seems to agree that we should do. I'm hoping that we'll publish starting with the 4.1 release.
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24165/#review52696
-----------------------------------------------------------
On July 31, 2014, 10:33 p.m., Robert Kanter wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24165/
> -----------------------------------------------------------
>
> (Updated July 31, 2014, 10:33 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1954
> https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> See https://issues.apache.org/jira/browse/OOZIE-1954
>
>
> Diffs
> -----
>
> client/src/main/resources/oozie-workflow-0.5.xsd 6620a4e
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 94b55cf
> core/src/test/java/org/apache/oozie/action/hadoop/MapperReducerForTest.java 7fe1f42
> docs/src/site/twiki/WorkflowFunctionalSpec.twiki 3319bcc
> examples/src/main/apps/map-reduce/job-with-config-class.properties PRE-CREATION
> examples/src/main/apps/map-reduce/job.properties 7b7a24c
> examples/src/main/apps/map-reduce/workflow-with-config-class.xml PRE-CREATION
> examples/src/main/java/org/apache/oozie/example/SampleOozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 2cb690f
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMapper.java 322f498
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/MapReduceMain.java f417a78
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfigurator.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorException.java PRE-CREATION
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/PipesMain.java 67a2a43
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/OozieActionConfiguratorForTest.java PRE-CREATION
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java e78c98a
> sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java bcc22f2
>
> Diff: https://reviews.apache.org/r/24165/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Robert Kanter
>
>