You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by pavan kumar kolamuri <pa...@gmail.com> on 2014/11/11 07:35:51 UTC
Re: Review Request 27614: OOZIE-1983 Add spark action executor in
oozie
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27614/
-----------------------------------------------------------
(Updated Nov. 11, 2014, 6:35 a.m.)
Review request for oozie and shwethags.
Summary (updated)
-----------------
OOZIE-1983 Add spark action executor in oozie
Bugs: OOZIE-1983
https://issues.apache.org/jira/browse/OOZIE-1983
Repository: oozie-git
Description
-------
Add spark action executor in oozie. Spark jobs can be run using oozie
Diffs
-----
client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b
client/src/main/resources/spark-action-0.1.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 42f2965
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 7349d3f
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java PRE-CREATION
core/src/main/resources/oozie-default.xml 17155a1
pom.xml 1e79186
sharelib/pom.xml aa479a8
sharelib/spark/pom.xml PRE-CREATION
sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java PRE-CREATION
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java PRE-CREATION
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java PRE-CREATION
src/main/assemblies/sharelib.xml 4a46b90
webapp/pom.xml 35776c5
Diff: https://reviews.apache.org/r/27614/diff/
Testing
-------
Both unit testing and end to end testing done
Thanks,
pavan kumar kolamuri
Re: Review Request 27614: OOZIE-1983 Add spark action executor in
oozie
Posted by pavan kumar kolamuri <pa...@gmail.com>.
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1483
> > <https://reviews.apache.org/r/27614/diff/2/?file=757470#file757470line1483>
> >
> > type should be the second argument. i.e.
> >
> > LOG.debug("class for [{0}] Action: [{1}]", type, function);
Sure will fix it
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, line 32
> > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line32>
> >
> > Can we name this something else? It's easy to confuse it with LauncherMapper.CONF_OOZIE_ACTION_MAIN_CLASS
This is not required any more, i will call SparkSubmit directly instead of using reflection
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, line 34
> > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line34>
> >
> > Do we actually need to use this? In my experience, it causes nothing but trouble...
Yes it is needed due to some conflicts in dependencies
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, lines 35-41
> > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line35>
> >
> > These should have more unique names (like we do in the other actions). This helps prevent collisions as they go into the actionConf.
> >
> > e.g.
> > oozie.spark.spark
> > oozie.spark.master
> > oozie.spark.mode
> > etc
Sure will do
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, line 78
> > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line78>
> >
> > We have this for DistCp because the classes are different between different versions of MapReduce and there's also two versions of DistCp. I'm not familiar enough with Spark, but is there any reason for a user to use a different Spark class?
> >
> > If not, then you can get rid of this to simplify things. In fact, SparkMain should subclass LauncherMain instead of JavaMain, and you can call SparkSubmit.main(...) directly instead of using reflection. Look at HiveMain or SqoopMain for examples.
I thought later somenbody can change the Spark job launcher class , so that we can take it from properties like distcp. Anyway i will remove it
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, line 85
> > <https://reviews.apache.org/r/27614/diff/2/?file=757471#file757471line85>
> >
> > Do we actually need to use this property? In my experience, it causes nothing but trouble...
Yes it is needed
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > sharelib/spark/pom.xml, line 41
> > <https://reviews.apache.org/r/27614/diff/2/?file=757475#file757475line41>
> >
> > whitespace
will fix
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > sharelib/spark/pom.xml, line 45
> > <https://reviews.apache.org/r/27614/diff/2/?file=757475#file757475line45>
> >
> > version numbers should go in the root pom
All other actions are using guava 11.0.2 , to run spark action we need 14.0.1 for the reason i dont want to change in root pom
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > sharelib/spark/pom.xml, line 51
> > <https://reviews.apache.org/r/27614/diff/2/?file=757475#file757475line51>
> >
> > versions numbers should go in the root pom
will do
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java, line 29
> > <https://reviews.apache.org/r/27614/diff/2/?file=757476#file757476line29>
> >
> > See earlier comment where I explain that this should subclass LauncherMain and other changes
Sure will change
> On Nov. 12, 2014, 8:17 a.m., Robert Kanter wrote:
> > sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java, line 100
> > <https://reviews.apache.org/r/27614/diff/2/?file=757476#file757476line100>
> >
> > Most of the other actions have some additional logic to send information back to Oozie (e.g. launched child job ids, other stats). I'm not sure but I believe Spark jobs have an ID, right? Is it possible to return this back to Oozie as the child job? Look at some of the other actions to see how we do this.
Yes spark jobs have application id when they are running in spark cluster , according to my knowledge i don't think there is way to get the application id from the cluster .
- pavan kumar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27614/#review60961
-----------------------------------------------------------
On Nov. 11, 2014, 6:35 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27614/
> -----------------------------------------------------------
>
> (Updated Nov. 11, 2014, 6:35 a.m.)
>
>
> Review request for oozie and shwethags.
>
>
> Bugs: OOZIE-1983
> https://issues.apache.org/jira/browse/OOZIE-1983
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Add spark action executor in oozie. Spark jobs can be run using oozie
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b
> client/src/main/resources/spark-action-0.1.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 42f2965
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 7349d3f
> core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java PRE-CREATION
> core/src/main/resources/oozie-default.xml 17155a1
> pom.xml 1e79186
> sharelib/pom.xml aa479a8
> sharelib/spark/pom.xml PRE-CREATION
> sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java PRE-CREATION
> src/main/assemblies/sharelib.xml 4a46b90
> webapp/pom.xml 35776c5
>
> Diff: https://reviews.apache.org/r/27614/diff/
>
>
> Testing
> -------
>
> Both unit testing and end to end testing done
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 27614: OOZIE-1983 Add spark action executor in
oozie
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27614/#review60961
-----------------------------------------------------------
core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/27614/#comment102383>
type should be the second argument. i.e.
LOG.debug("class for [{0}] Action: [{1}]", type, function);
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
<https://reviews.apache.org/r/27614/#comment102385>
Can we name this something else? It's easy to confuse it with LauncherMapper.CONF_OOZIE_ACTION_MAIN_CLASS
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
<https://reviews.apache.org/r/27614/#comment102384>
Do we actually need to use this? In my experience, it causes nothing but trouble...
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
<https://reviews.apache.org/r/27614/#comment102389>
These should have more unique names (like we do in the other actions). This helps prevent collisions as they go into the actionConf.
e.g.
oozie.spark.spark
oozie.spark.master
oozie.spark.mode
etc
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
<https://reviews.apache.org/r/27614/#comment102387>
We have this for DistCp because the classes are different between different versions of MapReduce and there's also two versions of DistCp. I'm not familiar enough with Spark, but is there any reason for a user to use a different Spark class?
If not, then you can get rid of this to simplify things. In fact, SparkMain should subclass LauncherMain instead of JavaMain, and you can call SparkSubmit.main(...) directly instead of using reflection. Look at HiveMain or SqoopMain for examples.
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
<https://reviews.apache.org/r/27614/#comment102388>
Do we actually need to use this property? In my experience, it causes nothing but trouble...
sharelib/spark/pom.xml
<https://reviews.apache.org/r/27614/#comment102390>
whitespace
sharelib/spark/pom.xml
<https://reviews.apache.org/r/27614/#comment102391>
version numbers should go in the root pom
sharelib/spark/pom.xml
<https://reviews.apache.org/r/27614/#comment102392>
versions numbers should go in the root pom
sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java
<https://reviews.apache.org/r/27614/#comment102393>
See earlier comment where I explain that this should subclass LauncherMain and other changes
sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java
<https://reviews.apache.org/r/27614/#comment102394>
Most of the other actions have some additional logic to send information back to Oozie (e.g. launched child job ids, other stats). I'm not sure but I believe Spark jobs have an ID, right? Is it possible to return this back to Oozie as the child job? Look at some of the other actions to see how we do this.
- Robert Kanter
On Nov. 11, 2014, 6:35 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27614/
> -----------------------------------------------------------
>
> (Updated Nov. 11, 2014, 6:35 a.m.)
>
>
> Review request for oozie and shwethags.
>
>
> Bugs: OOZIE-1983
> https://issues.apache.org/jira/browse/OOZIE-1983
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Add spark action executor in oozie. Spark jobs can be run using oozie
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b
> client/src/main/resources/spark-action-0.1.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 42f2965
> core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 7349d3f
> core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java PRE-CREATION
> core/src/main/resources/oozie-default.xml 17155a1
> pom.xml 1e79186
> sharelib/pom.xml aa479a8
> sharelib/spark/pom.xml PRE-CREATION
> sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java PRE-CREATION
> src/main/assemblies/sharelib.xml 4a46b90
> webapp/pom.xml 35776c5
>
> Diff: https://reviews.apache.org/r/27614/diff/
>
>
> Testing
> -------
>
> Both unit testing and end to end testing done
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 27614: OOZIE-1983 Add spark action executor in
oozie
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27614/#review62209
-----------------------------------------------------------
Ship it!
Ship It!
- Robert Kanter
On Nov. 19, 2014, 5:43 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27614/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2014, 5:43 a.m.)
>
>
> Review request for oozie and shwethags.
>
>
> Bugs: OOZIE-1983
> https://issues.apache.org/jira/browse/OOZIE-1983
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Add spark action executor in oozie. Spark jobs can be run using oozie
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b
> client/src/main/resources/spark-action-0.1.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java PRE-CREATION
> core/src/main/resources/oozie-default.xml 17155a1
> docs/src/site/twiki/DG_SparkActionExtension.twiki PRE-CREATION
> docs/src/site/twiki/index.twiki c8ba742
> pom.xml 1e79186
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java 8b8135a
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 8cfefbb
> sharelib/pom.xml aa479a8
> sharelib/spark/pom.xml PRE-CREATION
> sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java PRE-CREATION
> src/main/assemblies/sharelib.xml 4a46b90
> webapp/pom.xml 35776c5
>
> Diff: https://reviews.apache.org/r/27614/diff/
>
>
> Testing
> -------
>
> Both unit testing and end to end testing done
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 27614: OOZIE-1983 Add spark action executor in
oozie
Posted by pavan kumar kolamuri <pa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27614/
-----------------------------------------------------------
(Updated Nov. 19, 2014, 5:43 a.m.)
Review request for oozie and shwethags.
Changes
-------
Fixed based on comments .
Bugs: OOZIE-1983
https://issues.apache.org/jira/browse/OOZIE-1983
Repository: oozie-git
Description
-------
Add spark action executor in oozie. Spark jobs can be run using oozie
Diffs (updated)
-----
client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b
client/src/main/resources/spark-action-0.1.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java PRE-CREATION
core/src/main/resources/oozie-default.xml 17155a1
docs/src/site/twiki/DG_SparkActionExtension.twiki PRE-CREATION
docs/src/site/twiki/index.twiki c8ba742
pom.xml 1e79186
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java 8b8135a
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 8cfefbb
sharelib/pom.xml aa479a8
sharelib/spark/pom.xml PRE-CREATION
sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java PRE-CREATION
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java PRE-CREATION
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java PRE-CREATION
src/main/assemblies/sharelib.xml 4a46b90
webapp/pom.xml 35776c5
Diff: https://reviews.apache.org/r/27614/diff/
Testing
-------
Both unit testing and end to end testing done
Thanks,
pavan kumar kolamuri
Re: Review Request 27614: OOZIE-1983 Add spark action executor in
oozie
Posted by pavan kumar kolamuri <pa...@gmail.com>.
> On Nov. 19, 2014, 2:01 a.m., Robert Kanter wrote:
> > Looks good overall. I just had some minor, mostly trivial, comments this time.
> >
> > I'm sure you've done some testing. Have you made sure that a Spark job that fails is properly handled?
Yes i have tested end to end , yes it was handled when spark job fails even the workflow will fail.
> On Nov. 19, 2014, 2:01 a.m., Robert Kanter wrote:
> > core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java, lines 85-86
> > <https://reviews.apache.org/r/27614/diff/3/?file=761959#file761959line85>
> >
> > Please use curly braces around the if statement. It's very easy for someone later to accidently mess something up here otherwise
Ok
> On Nov. 19, 2014, 2:01 a.m., Robert Kanter wrote:
> > sharelib/spark/pom.xml, lines 31-32
> > <https://reviews.apache.org/r/27614/diff/3/?file=761967#file761967line31>
> >
> > I know this is silly, but capitalize 's' in Spark
Done
> On Nov. 19, 2014, 2:01 a.m., Robert Kanter wrote:
> > sharelib/spark/pom.xml, line 45
> > <https://reviews.apache.org/r/27614/diff/3/?file=761967#file761967line45>
> >
> > In that case, can we add a property to the root pom so we can set the version there and reference it here? That way we're less likely to forget about this in the future.
> >
> > e.g.
> > ${spark.guava.version}
Sure will use this
> On Nov. 19, 2014, 2:01 a.m., Robert Kanter wrote:
> > sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java, line 45
> > <https://reviews.apache.org/r/27614/diff/3/?file=761968#file761968line45>
> >
> > Can you add the below print statement here like we do in most of the other actions?
> >
> > e.g.
> > System.out.println();
> > System.out.println("Oozie Spark action configuration");
> > System.out.println("=================================================================");
> > System.out.println();
will add
> On Nov. 19, 2014, 2:01 a.m., Robert Kanter wrote:
> > sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java, line 93
> > <https://reviews.apache.org/r/27614/diff/3/?file=761968#file761968line93>
> >
> > Can you add the below print statement here like we do in most of the other actions?
> >
> > System.out.println("=================================================================");
> > System.out.println();
> > System.out.println(">>> Invoking Spark class now >>>");
> > System.out.println();
> > System.out.flush();
will add
- pavan kumar
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27614/#review62107
-----------------------------------------------------------
On Nov. 13, 2014, 6:49 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27614/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2014, 6:49 a.m.)
>
>
> Review request for oozie and shwethags.
>
>
> Bugs: OOZIE-1983
> https://issues.apache.org/jira/browse/OOZIE-1983
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Add spark action executor in oozie. Spark jobs can be run using oozie
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b
> client/src/main/resources/spark-action-0.1.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java PRE-CREATION
> core/src/main/resources/oozie-default.xml 17155a1
> docs/src/site/twiki/DG_SparkActionExtension.twiki PRE-CREATION
> docs/src/site/twiki/index.twiki c8ba742
> pom.xml 1e79186
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java 8b8135a
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 8cfefbb
> sharelib/pom.xml aa479a8
> sharelib/spark/pom.xml PRE-CREATION
> sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java PRE-CREATION
> src/main/assemblies/sharelib.xml 4a46b90
> webapp/pom.xml 35776c5
>
> Diff: https://reviews.apache.org/r/27614/diff/
>
>
> Testing
> -------
>
> Both unit testing and end to end testing done
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 27614: OOZIE-1983 Add spark action executor in
oozie
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27614/#review62107
-----------------------------------------------------------
Looks good overall. I just had some minor, mostly trivial, comments this time.
I'm sure you've done some testing. Have you made sure that a Spark job that fails is properly handled?
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java
<https://reviews.apache.org/r/27614/#comment104050>
Please use curly braces around the if statement. It's very easy for someone later to accidently mess something up here otherwise
sharelib/spark/pom.xml
<https://reviews.apache.org/r/27614/#comment104051>
I know this is silly, but capitalize 's' in Spark
sharelib/spark/pom.xml
<https://reviews.apache.org/r/27614/#comment104052>
In that case, can we add a property to the root pom so we can set the version there and reference it here? That way we're less likely to forget about this in the future.
e.g.
${spark.guava.version}
sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java
<https://reviews.apache.org/r/27614/#comment104053>
Can you add the below print statement here like we do in most of the other actions?
e.g.
System.out.println();
System.out.println("Oozie Spark action configuration");
System.out.println("=================================================================");
System.out.println();
sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java
<https://reviews.apache.org/r/27614/#comment104054>
Can you add the below print statement here like we do in most of the other actions?
System.out.println("=================================================================");
System.out.println();
System.out.println(">>> Invoking Spark class now >>>");
System.out.println();
System.out.flush();
- Robert Kanter
On Nov. 13, 2014, 6:49 a.m., pavan kumar kolamuri wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27614/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2014, 6:49 a.m.)
>
>
> Review request for oozie and shwethags.
>
>
> Bugs: OOZIE-1983
> https://issues.apache.org/jira/browse/OOZIE-1983
>
>
> Repository: oozie-git
>
>
> Description
> -------
>
> Add spark action executor in oozie. Spark jobs can be run using oozie
>
>
> Diffs
> -----
>
> client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b
> client/src/main/resources/spark-action-0.1.xsd PRE-CREATION
> core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java PRE-CREATION
> core/src/main/resources/oozie-default.xml 17155a1
> docs/src/site/twiki/DG_SparkActionExtension.twiki PRE-CREATION
> docs/src/site/twiki/index.twiki c8ba742
> pom.xml 1e79186
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java 8b8135a
> sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 8cfefbb
> sharelib/pom.xml aa479a8
> sharelib/spark/pom.xml PRE-CREATION
> sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java PRE-CREATION
> sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java PRE-CREATION
> src/main/assemblies/sharelib.xml 4a46b90
> webapp/pom.xml 35776c5
>
> Diff: https://reviews.apache.org/r/27614/diff/
>
>
> Testing
> -------
>
> Both unit testing and end to end testing done
>
>
> Thanks,
>
> pavan kumar kolamuri
>
>
Re: Review Request 27614: OOZIE-1983 Add spark action executor in
oozie
Posted by pavan kumar kolamuri <pa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27614/
-----------------------------------------------------------
(Updated Nov. 13, 2014, 6:49 a.m.)
Review request for oozie and shwethags.
Changes
-------
Add documentation, made changes based on comments
Bugs: OOZIE-1983
https://issues.apache.org/jira/browse/OOZIE-1983
Repository: oozie-git
Description
-------
Add spark action executor in oozie. Spark jobs can be run using oozie
Diffs (updated)
-----
client/src/main/java/org/apache/oozie/cli/OozieCLI.java 9c2d14b
client/src/main/resources/spark-action-0.1.xsd PRE-CREATION
core/src/main/java/org/apache/oozie/action/hadoop/SparkActionExecutor.java PRE-CREATION
core/src/main/resources/oozie-default.xml 17155a1
docs/src/site/twiki/DG_SparkActionExtension.twiki PRE-CREATION
docs/src/site/twiki/index.twiki c8ba742
pom.xml 1e79186
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/JavaMain.java 8b8135a
sharelib/oozie/src/main/java/org/apache/oozie/action/hadoop/LauncherMain.java 8cfefbb
sharelib/pom.xml aa479a8
sharelib/spark/pom.xml PRE-CREATION
sharelib/spark/src/main/java/org.apache.oozie.action.hadoop/SparkMain.java PRE-CREATION
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkActionExecutor.java PRE-CREATION
sharelib/spark/src/test/java/org/apache/oozie/action/hadoop/TestSparkMain.java PRE-CREATION
src/main/assemblies/sharelib.xml 4a46b90
webapp/pom.xml 35776c5
Diff: https://reviews.apache.org/r/27614/diff/
Testing
-------
Both unit testing and end to end testing done
Thanks,
pavan kumar kolamuri