You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Alejandro Abdelnur <tu...@cloudera.com> on 2012/05/08 08:28:39 UTC
Review Request: OOZIE-830 add support for multiple/configurable sharelibs
for each action type
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5063/
-----------------------------------------------------------
Review request for oozie.
Summary
-------
The sharelibrary override is supported at Oozie server level and at workflow job level.
The share library name will be resolved using the following precedence order:
action.sharelib.for.#ACTIONTYPE# in the action configuration (at action level)
action.sharelib.for.#ACTIONTYPE# in the job configuration (at job level)
action.sharelib.for.#ACTIONTYPE# in the oozie server configuration (at oozie server level)
action's ActionExecutor getDefaultShareLibName() method (default)
This addresses bug OOZIE-830.
https://issues.apache.org/jira/browse/OOZIE-830
Diffs
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1335346
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1335346
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1335346
trunk/docs/src/site/twiki/WorkflowFunctionalSpec.twiki 1335346
Diff: https://reviews.apache.org/r/5063/diff
Testing
-------
Thanks,
Alejandro
Re: Review Request: OOZIE-830 add support for multiple/configurable
sharelibs for each action type
Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5063/#review7707
-----------------------------------------------------------
Looks good...some comments mainly related to javadoc
trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java
<https://reviews.apache.org/r/5063/#comment16977>
javadoc missing
trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java
<https://reviews.apache.org/r/5063/#comment16979>
Does this sentence need modification as the function name has changed?.. Same comment at other places
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/5063/#comment16983>
do you want to remove 'postfix' from method signature here too?
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/5063/#comment16980>
why is this function called twice? here and in submitLauncher
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/5063/#comment16981>
missing javadoc
trunk/docs/src/site/twiki/WorkflowFunctionalSpec.twiki
<https://reviews.apache.org/r/5063/#comment16982>
'.' after distcp actions
- Virag
On 2012-05-08 07:06:23, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5063/
> -----------------------------------------------------------
>
> (Updated 2012-05-08 07:06:23)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The sharelibrary override is supported at Oozie server level and at workflow job level.
>
> The share library name will be resolved using the following precedence order:
>
> action.sharelib.for.#ACTIONTYPE# in the action configuration (at action level)
> action.sharelib.for.#ACTIONTYPE# in the job configuration (at job level)
> action.sharelib.for.#ACTIONTYPE# in the oozie server configuration (at oozie server level)
> action's ActionExecutor getDefaultShareLibName() method (default)
>
>
> This addresses bug OOZIE-830.
> https://issues.apache.org/jira/browse/OOZIE-830
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1335346
> trunk/docs/src/site/twiki/WorkflowFunctionalSpec.twiki 1335346
>
> Diff: https://reviews.apache.org/r/5063/diff
>
>
> Testing
> -------
>
> besides testcases, tested in a deployed setup
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-830 add support for multiple/configurable
sharelibs for each action type
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
> On 2012-05-09 20:35:13, Mohammad Islam wrote:
> > How will a user know what are the system supported versions? There could be a utility for that.
> >
> > Another point, if a user gives a wrong or not supported version, how will that be communicated to the end-user.
> >
listing the contents of the HDFS /user/oozie/share/lib directory will tell the supported versions.
a wrong version will make the action to fail with JARs not found, if the dir is not avail in HDFS the action won't have the JARs files in its classpath
> On 2012-05-09 20:35:13, Mohammad Islam wrote:
> > trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 1116
> > <https://reviews.apache.org/r/5063/diff/3/?file=107695#file107695line1116>
> >
> > If name is 'null' after that, what will happen?
> > Should we throw an error?
> >
> >
no, if null it means the action does not have a sharelib (ie mapreduce or java)
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5063/#review7741
-----------------------------------------------------------
On 2012-05-08 07:06:23, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5063/
> -----------------------------------------------------------
>
> (Updated 2012-05-08 07:06:23)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The sharelibrary override is supported at Oozie server level and at workflow job level.
>
> The share library name will be resolved using the following precedence order:
>
> action.sharelib.for.#ACTIONTYPE# in the action configuration (at action level)
> action.sharelib.for.#ACTIONTYPE# in the job configuration (at job level)
> action.sharelib.for.#ACTIONTYPE# in the oozie server configuration (at oozie server level)
> action's ActionExecutor getDefaultShareLibName() method (default)
>
>
> This addresses bug OOZIE-830.
> https://issues.apache.org/jira/browse/OOZIE-830
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1335346
> trunk/docs/src/site/twiki/WorkflowFunctionalSpec.twiki 1335346
>
> Diff: https://reviews.apache.org/r/5063/diff
>
>
> Testing
> -------
>
> besides testcases, tested in a deployed setup
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-830 add support for multiple/configurable
sharelibs for each action type
Posted by Mohammad Islam <mi...@yahoo.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5063/#review7741
-----------------------------------------------------------
How will a user know what are the system supported versions? There could be a utility for that.
Another point, if a user gives a wrong or not supported version, how will that be communicated to the end-user.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/5063/#comment17041>
If name is 'null' after that, what will happen?
Should we throw an error?
- Mohammad
On 2012-05-08 07:06:23, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5063/
> -----------------------------------------------------------
>
> (Updated 2012-05-08 07:06:23)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The sharelibrary override is supported at Oozie server level and at workflow job level.
>
> The share library name will be resolved using the following precedence order:
>
> action.sharelib.for.#ACTIONTYPE# in the action configuration (at action level)
> action.sharelib.for.#ACTIONTYPE# in the job configuration (at job level)
> action.sharelib.for.#ACTIONTYPE# in the oozie server configuration (at oozie server level)
> action's ActionExecutor getDefaultShareLibName() method (default)
>
>
> This addresses bug OOZIE-830.
> https://issues.apache.org/jira/browse/OOZIE-830
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1335346
> trunk/docs/src/site/twiki/WorkflowFunctionalSpec.twiki 1335346
>
> Diff: https://reviews.apache.org/r/5063/diff
>
>
> Testing
> -------
>
> besides testcases, tested in a deployed setup
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-830 add support for multiple/configurable
sharelibs for each action type
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5063/#review7749
-----------------------------------------------------------
i'll prepare a patch addressing Virag's corrections.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/5063/#comment17052>
here is injected into the launcherconf, in submitLauncher is injected into the actionconf.
- Alejandro
On 2012-05-08 07:06:23, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5063/
> -----------------------------------------------------------
>
> (Updated 2012-05-08 07:06:23)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The sharelibrary override is supported at Oozie server level and at workflow job level.
>
> The share library name will be resolved using the following precedence order:
>
> action.sharelib.for.#ACTIONTYPE# in the action configuration (at action level)
> action.sharelib.for.#ACTIONTYPE# in the job configuration (at job level)
> action.sharelib.for.#ACTIONTYPE# in the oozie server configuration (at oozie server level)
> action's ActionExecutor getDefaultShareLibName() method (default)
>
>
> This addresses bug OOZIE-830.
> https://issues.apache.org/jira/browse/OOZIE-830
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1335346
> trunk/docs/src/site/twiki/WorkflowFunctionalSpec.twiki 1335346
>
> Diff: https://reviews.apache.org/r/5063/diff
>
>
> Testing
> -------
>
> besides testcases, tested in a deployed setup
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-830 add support for multiple/configurable
sharelibs for each action type
Posted by Mayank Bansal <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5063/#review7739
-----------------------------------------------------------
Ship it!
+1 looks good after virag's comments
- Mayank
On 2012-05-08 07:06:23, Alejandro Abdelnur wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5063/
> -----------------------------------------------------------
>
> (Updated 2012-05-08 07:06:23)
>
>
> Review request for oozie.
>
>
> Summary
> -------
>
> The sharelibrary override is supported at Oozie server level and at workflow job level.
>
> The share library name will be resolved using the following precedence order:
>
> action.sharelib.for.#ACTIONTYPE# in the action configuration (at action level)
> action.sharelib.for.#ACTIONTYPE# in the job configuration (at job level)
> action.sharelib.for.#ACTIONTYPE# in the oozie server configuration (at oozie server level)
> action's ActionExecutor getDefaultShareLibName() method (default)
>
>
> This addresses bug OOZIE-830.
> https://issues.apache.org/jira/browse/OOZIE-830
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1335346
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1335346
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1335346
> trunk/docs/src/site/twiki/WorkflowFunctionalSpec.twiki 1335346
>
> Diff: https://reviews.apache.org/r/5063/diff
>
>
> Testing
> -------
>
> besides testcases, tested in a deployed setup
>
>
> Thanks,
>
> Alejandro
>
>
Re: Review Request: OOZIE-830 add support for multiple/configurable
sharelibs for each action type
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5063/
-----------------------------------------------------------
(Updated 2012-05-08 07:06:23.568711)
Review request for oozie.
Summary
-------
The sharelibrary override is supported at Oozie server level and at workflow job level.
The share library name will be resolved using the following precedence order:
action.sharelib.for.#ACTIONTYPE# in the action configuration (at action level)
action.sharelib.for.#ACTIONTYPE# in the job configuration (at job level)
action.sharelib.for.#ACTIONTYPE# in the oozie server configuration (at oozie server level)
action's ActionExecutor getDefaultShareLibName() method (default)
This addresses bug OOZIE-830.
https://issues.apache.org/jira/browse/OOZIE-830
Diffs
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1335346
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1335346
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1335346
trunk/docs/src/site/twiki/WorkflowFunctionalSpec.twiki 1335346
Diff: https://reviews.apache.org/r/5063/diff
Testing (updated)
-------
besides testcases, tested in a deployed setup
Thanks,
Alejandro
Re: Review Request: OOZIE-830 add support for multiple/configurable
sharelibs for each action type
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5063/
-----------------------------------------------------------
(Updated 2012-05-08 06:29:52.868583)
Review request for oozie.
Changes
-------
removing trailing spaces
Summary
-------
The sharelibrary override is supported at Oozie server level and at workflow job level.
The share library name will be resolved using the following precedence order:
action.sharelib.for.#ACTIONTYPE# in the action configuration (at action level)
action.sharelib.for.#ACTIONTYPE# in the job configuration (at job level)
action.sharelib.for.#ACTIONTYPE# in the oozie server configuration (at oozie server level)
action's ActionExecutor getDefaultShareLibName() method (default)
This addresses bug OOZIE-830.
https://issues.apache.org/jira/browse/OOZIE-830
Diffs (updated)
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1335346
trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1335346
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1335346
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1335346
trunk/docs/src/site/twiki/WorkflowFunctionalSpec.twiki 1335346
Diff: https://reviews.apache.org/r/5063/diff
Testing
-------
Thanks,
Alejandro