You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by "Alejandro Abdelnur (Created) (JIRA)" <ji...@apache.org> on 2011/11/22 09:00:41 UTC
[jira] [Created] (OOZIE-610) Oozie system share lib should have
jars per action type
Oozie system share lib should have jars per action type
-------------------------------------------------------
Key: OOZIE-610
URL: https://issues.apache.org/jira/browse/OOZIE-610
Project: Oozie
Issue Type: Improvement
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
This creates a few issues:
* The classpath for an action grows significantly
* For a given action only a portion of the classpath is useful, the rest is dead code
* As more actions are added, the chances of conflicting dependencies grows.
As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
share/lib/pig/*
share/lib/streaming/*
share/lib/hive/*
share/lib/sqoop/*
Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155572#comment-13155572 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > Overall, it's an important problem.
bq. >
bq. > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
bq. >
bq. > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.
bq. >
bq. >
Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.
Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.
Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java, line 30
bq. > <https://reviews.apache.org/r/2909/diff/3/?file=59834#file59834line30>
bq. >
bq. > do we need it?
yes, because of the signature of the getShareLibPostFix() method
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java, line 209
bq. > <https://reviews.apache.org/r/2909/diff/3/?file=59836#file59836line209>
bq. >
bq. > so the name will be defined in each action. Or will it be better in the system configuration in oozie-default(site).xml?
this libraries are specific to an action type, thus it belongs to the action. Making this configurable does not seem necessary at this point.
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java, line 140
bq. > <https://reviews.apache.org/r/2909/diff/3/?file=59837#file59837line140>
bq. >
bq. > Is hard-coding a good approach? Default could be "pig".
i'd start simple, later we can add more intelligence/configurability if we find it necessary, as it is in a method already, the change will be transparent if we do it.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155663#comment-13155663 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3459
-----------------------------------------------------------
Overall comments:
1. Doing it incrementally will be good idea. But we need to consider the backward compatibility. For example, currently we are reading pig/*.jar, in the next oozie release, we decided to support pig/lib/*.jar or pig/0.8/lib/*.jar or pig/stable/lib/*.jar. In this case, the old setup will not work.
2. We wanted to support multiple version too. Virag has the patch. What is the best way of getting the benefits of both?
/trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java
<https://reviews.apache.org/r/2909/#comment7713>
But I don't see any other code changes in this file.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/2909/#comment7716>
Putting this condition earlier makes more sense.
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
<https://reviews.apache.org/r/2909/#comment7715>
it's ok
/trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java
<https://reviews.apache.org/r/2909/#comment7714>
why are we not reading the system library path for jars. There could be some system jars ,that are not any product specific (like pig, hive) but are commonly used. Also the question of backward compatibility is there.
/trunk/docs/src/site/twiki/DG_Examples.twiki
<https://reviews.apache.org/r/2909/#comment7712>
So user level libpath will not be supported?
- Mohammad
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "Alejandro Abdelnur (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur updated OOZIE-610:
-------------------------------------
Fix Version/s: 3.2.0
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Fix For: 3.2.0
>
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156263#comment-13156263 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > Overall, it's an important problem.
bq. >
bq. > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
bq. >
bq. > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.
bq.
bq. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.
bq.
bq. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.
bq.
bq. Alejandro Abdelnur wrote:
bq. I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.
bq.
bq. Thanks.
bq.
bq. Mohammad Islam wrote:
bq. I think the "use_version" support is considered here.
bq. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2
bq. Please review this patch.
bq.
bq. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)
bq. https://issues.apache.org/jira/browse/OOZIE-555
bq.
bq.
bq.
bq.
bq. Alejandro Abdelnur wrote:
bq. Mmmh (regarding my lost comments).
bq.
bq. OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)
bq.
bq. To achieve similar functionality on top of OOZIE-610 we could:
bq.
bq. * Change the sharelib path to contain action library versions, ie:
bq.
bq. share/lib/streaming/0.20.204.0/
bq. share/lib/pig/0.8.0/
bq. share/lib/pig/0.9.0/
bq.
bq. * Have an oozie-site configuration, per action that defines the stable version of the library, ie:
bq.
bq. oozie.action.stable.library.version.streaming=0.20.204.0
bq. oozie.action.stable.library.version.pig=0.8.0
bq.
bq. * The ActionExecutor getShareLibPostFix() method would have the following logic:
bq.
bq. getShareLibPostFix(...) {
bq.
bq. String version = action.config.get("oozie.action.library.version");
bq. if (version == null) {
bq. version = oozie.config.get("oozie.action.stable.library.version");
bq. }
bq. if (version != null) {
bq. //construct the library path using the specified version
bq. }
bq. }
bq.
bq. This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.
bq.
bq.
bq.
bq. If we are OK with this approach I'll file a JIRA with it.
bq.
bq.
bq. Mohammad Islam wrote:
bq. In addition to your proposal, we want to give the user the flexibility to choose the version *optionally*. User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version.
bq.
bq. there is already a JIRA on this:
bq. https://issues.apache.org/jira/browse/OOZIE-555
bq. You can add your comments there.
bq.
bq. Alejandro Abdelnur wrote:
bq. In my proposal, if the user sets the configuration property 'oozie.action.library.version' to any value, that version will be used. If the user does not set it the default stable one will be used.
bq.
bq. I'll put my proposal in OOZIE-555
The end-user who writes the workflow.xml should have a way of controlling his version. For example, one user tested his pig script againist pig 0.8 another one has used the latest (0.9). We should allow the both users to support that provide pig 0.8 and pig 0.9 are supported versions. I think it should be defined per-action level.
Also we had the same design discussion few months back.
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156317#comment-13156317 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/
-----------------------------------------------------------
(Updated 2011-11-23 21:45:14.340237)
Review request for oozie.
Changes
-------
removing spurious import from ActionExecutor.java
Summary
-------
breaks the Oozie sharelib into one subdir per action type
This addresses bug OOZIE-610.
https://issues.apache.org/jira/browse/OOZIE-610
Diffs (updated)
-----
/trunk/core/pom.xml 1205616
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205616
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205616
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205616
/trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205616
/trunk/docs/src/site/twiki/DG_Examples.twiki 1205616
/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205616
/trunk/examples/pom.xml 1205616
/trunk/examples/src/main/apps/custom-main/job.properties 1205616
/trunk/examples/src/main/apps/demo/job.properties 1205616
/trunk/examples/src/main/apps/hadoop-el/job.properties 1205616
/trunk/examples/src/main/apps/java-main/job.properties 1205616
/trunk/examples/src/main/apps/pig/job.properties 1205616
/trunk/examples/src/main/apps/streaming/job.properties 1205616
/trunk/pom.xml 1205616
/trunk/sharelib/pig/pom.xml PRE-CREATION
/trunk/sharelib/pom.xml 1205616
/trunk/sharelib/streaming/pom.xml PRE-CREATION
/trunk/src/main/assemblies/distro.xml 1205616
/trunk/src/main/assemblies/examples.xml 1205616
/trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
/trunk/src/main/assemblies/sharelib.xml 1205616
Diff: https://reviews.apache.org/r/2909/diff
Testing
-------
built, installed and run pig and streaming examples
Thanks,
Alejandro
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156229#comment-13156229 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > Overall, it's an important problem.
bq. >
bq. > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
bq. >
bq. > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.
bq.
bq. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.
bq.
bq. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.
bq.
bq. Alejandro Abdelnur wrote:
bq. I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.
bq.
bq. Thanks.
bq.
bq. Mohammad Islam wrote:
bq. I think the "use_version" support is considered here.
bq. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2
bq. Please review this patch.
bq.
bq. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)
bq. https://issues.apache.org/jira/browse/OOZIE-555
bq.
bq.
bq.
bq.
bq. Alejandro Abdelnur wrote:
bq. Mmmh (regarding my lost comments).
bq.
bq. OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)
bq.
bq. To achieve similar functionality on top of OOZIE-610 we could:
bq.
bq. * Change the sharelib path to contain action library versions, ie:
bq.
bq. share/lib/streaming/0.20.204.0/
bq. share/lib/pig/0.8.0/
bq. share/lib/pig/0.9.0/
bq.
bq. * Have an oozie-site configuration, per action that defines the stable version of the library, ie:
bq.
bq. oozie.action.stable.library.version.streaming=0.20.204.0
bq. oozie.action.stable.library.version.pig=0.8.0
bq.
bq. * The ActionExecutor getShareLibPostFix() method would have the following logic:
bq.
bq. getShareLibPostFix(...) {
bq.
bq. String version = action.config.get("oozie.action.library.version");
bq. if (version == null) {
bq. version = oozie.config.get("oozie.action.stable.library.version");
bq. }
bq. if (version != null) {
bq. //construct the library path using the specified version
bq. }
bq. }
bq.
bq. This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.
bq.
bq.
bq.
bq. If we are OK with this approach I'll file a JIRA with it.
bq.
In addition to your proposal, we want to give the user the flexibility to choose the version *optionally*. User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version.
there is already a JIRA on this:
https://issues.apache.org/jira/browse/OOZIE-555
You can add your comments there.
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "Alejandro Abdelnur (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154946#comment-13154946 ]
Alejandro Abdelnur commented on OOZIE-610:
------------------------------------------
Forgot to mention the different between this JIRA & OOZIE-552.
OOZIE-552 allows a user to specify multiple directories in the sharelib path (a la classpath), thus aggregating all JARs from those directories.
This JIRA selects the directory to add to the classpath based on the action type. When running streaming action only streaming JARs are added, when running pig action only pig JARs are added.
This JIRA only affects Oozie system sharelib, the functionality introduced by OOZIE-552 is still available. Furthermore they are complimentary.
Note that for the case of conflicting dependencies between Pig and Hive OOZIE-552 does not provide a solution as the the multiple directories paths are aggregated to all actions
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "Mohammad Kamrul Islam (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154943#comment-13154943 ]
Mohammad Kamrul Islam commented on OOZIE-610:
---------------------------------------------
This issue is being addressed into Jira OOZIE-552.
In addition, it supports multiple version.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155560#comment-13155560 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
Overall, it's an important problem.
In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.
/trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java
<https://reviews.apache.org/r/2909/#comment7693>
do we need it?
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java
<https://reviews.apache.org/r/2909/#comment7695>
so the name will be defined in each action. Or will it be better in the system configuration in oozie-default(site).xml?
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
<https://reviews.apache.org/r/2909/#comment7697>
Is hard-coding a good approach? Default could be "pig".
- Mohammad
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155431#comment-13155431 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/
-----------------------------------------------------------
(Updated 2011-11-22 21:03:19.865952)
Review request for oozie.
Changes
-------
rebased to trunk (OOZIE-565)
Summary
-------
breaks the Oozie sharelib into one subdir per action type
This addresses bug OOZIE-610.
https://issues.apache.org/jira/browse/OOZIE-610
Diffs (updated)
-----
/trunk/core/pom.xml 1205165
/trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
/trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
/trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
/trunk/examples/pom.xml 1205165
/trunk/examples/src/main/apps/custom-main/job.properties 1205165
/trunk/examples/src/main/apps/demo/job.properties 1205165
/trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
/trunk/examples/src/main/apps/java-main/job.properties 1205165
/trunk/examples/src/main/apps/pig/job.properties 1205165
/trunk/examples/src/main/apps/streaming/job.properties 1205165
/trunk/pom.xml 1205165
/trunk/sharelib/pig/pom.xml PRE-CREATION
/trunk/sharelib/pom.xml 1205165
/trunk/sharelib/streaming/pom.xml PRE-CREATION
/trunk/src/main/assemblies/distro.xml 1205165
/trunk/src/main/assemblies/examples.xml 1205165
/trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
/trunk/src/main/assemblies/sharelib.xml 1205165
Diff: https://reviews.apache.org/r/2909/diff
Testing
-------
built, installed and run pig and streaming examples
Thanks,
Alejandro
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155978#comment-13155978 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > Overall, it's an important problem.
bq. >
bq. > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
bq. >
bq. > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.
bq.
bq. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.
bq.
bq. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.
bq.
bq. Alejandro Abdelnur wrote:
bq. I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.
bq.
bq. Thanks.
bq.
bq. Mohammad Islam wrote:
bq. I think the "use_version" support is considered here.
bq. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2
bq. Please review this patch.
bq.
bq. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)
bq. https://issues.apache.org/jira/browse/OOZIE-555
bq.
bq.
bq.
Mmmh (regarding my lost comments).
OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)
To achieve similar functionality on top of OOZIE-610 we could:
* Change the sharelib path to contain action library versions, ie:
share/lib/streaming/0.20.204.0/
share/lib/pig/0.8.0/
share/lib/pig/0.9.0/
* Have an oozie-site configuration, per action that defines the stable version of the library, ie:
oozie.action.stable.library.version.streaming=0.20.204.0
oozie.action.stable.library.version.pig=0.8.0
* The ActionExecutor getShareLibPostFix() method would have the following logic:
getShareLibPostFix(...) {
String version = action.config.get("oozie.action.library.version");
if (version == null) {
version = oozie.config.get("oozie.action.stable.library.version");
}
if (version != null) {
//construct the library path using the specified version
}
}
This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.
If we are OK with this approach I'll file a JIRA with it.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155590#comment-13155590 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > Overall, it's an important problem.
bq. >
bq. > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
bq. >
bq. > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.
bq.
bq. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.
bq.
bq. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.
I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.
Thanks.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156241#comment-13156241 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > Overall, it's an important problem.
bq. >
bq. > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
bq. >
bq. > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.
bq.
bq. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.
bq.
bq. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.
bq.
bq. Alejandro Abdelnur wrote:
bq. I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.
bq.
bq. Thanks.
bq.
bq. Mohammad Islam wrote:
bq. I think the "use_version" support is considered here.
bq. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2
bq. Please review this patch.
bq.
bq. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)
bq. https://issues.apache.org/jira/browse/OOZIE-555
bq.
bq.
bq.
bq.
bq. Alejandro Abdelnur wrote:
bq. Mmmh (regarding my lost comments).
bq.
bq. OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)
bq.
bq. To achieve similar functionality on top of OOZIE-610 we could:
bq.
bq. * Change the sharelib path to contain action library versions, ie:
bq.
bq. share/lib/streaming/0.20.204.0/
bq. share/lib/pig/0.8.0/
bq. share/lib/pig/0.9.0/
bq.
bq. * Have an oozie-site configuration, per action that defines the stable version of the library, ie:
bq.
bq. oozie.action.stable.library.version.streaming=0.20.204.0
bq. oozie.action.stable.library.version.pig=0.8.0
bq.
bq. * The ActionExecutor getShareLibPostFix() method would have the following logic:
bq.
bq. getShareLibPostFix(...) {
bq.
bq. String version = action.config.get("oozie.action.library.version");
bq. if (version == null) {
bq. version = oozie.config.get("oozie.action.stable.library.version");
bq. }
bq. if (version != null) {
bq. //construct the library path using the specified version
bq. }
bq. }
bq.
bq. This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.
bq.
bq.
bq.
bq. If we are OK with this approach I'll file a JIRA with it.
bq.
bq.
bq. Mohammad Islam wrote:
bq. In addition to your proposal, we want to give the user the flexibility to choose the version *optionally*. User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version.
bq.
bq. there is already a JIRA on this:
bq. https://issues.apache.org/jira/browse/OOZIE-555
bq. You can add your comments there.
In my proposal, if the user sets the configuration property 'oozie.action.library.version' to any value, that version will be used. If the user does not set it the default stable one will be used.
I'll put my proposal in OOZIE-555
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155694#comment-13155694 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-23 03:31:31, Mohammad Islam wrote:
bq. > Overall comments:
bq. > 1. Doing it incrementally will be good idea. But we need to consider the backward compatibility. For example, currently we are reading pig/*.jar, in the next oozie release, we decided to support pig/lib/*.jar or pig/0.8/lib/*.jar or pig/stable/lib/*.jar. In this case, the old setup will not work.
bq. >
bq. > 2. We wanted to support multiple version too. Virag has the patch. What is the best way of getting the benefits of both?
bq. >
Backwards compatibility would not be an issue because when you change the layout of the library you'll change the resolution logic, and for jobs will be transparent. When you upgrade Oozie you'll have to upgrade the sharelib at the same time; but WF apps will be unaffected. Furthermore a migration path would be that now you are only supporting one version, 'stable', and later you'll support other versions via a configuration in the action XML.
Alternate versions of libraries could be supported by a 'oozie.action.library.version' property in the action configuration, if not set it would resolve to 'stable', if set it would resolve to the version specified.
This change is not complex, actually it is quite simple. What complicates things a bit is the packaging of sharelibs.
Again, we could do this incremental (as in a couple of weeks), I just don't want to delay this patch much more as I have a patch chain (this, Hive, Sqoop, Alfredo) which takes time and work to keep up to date.
bq. On 2011-11-23 03:31:31, Mohammad Islam wrote:
bq. > /trunk/docs/src/site/twiki/DG_Examples.twiki, line 95
bq. > <https://reviews.apache.org/r/2909/diff/3/?file=59839#file59839line95>
bq. >
bq. > So user level libpath will not be supported?
This is still supported, the examples are not using it anymore, instead they use the Oozie share lib.
bq. On 2011-11-23 03:31:31, Mohammad Islam wrote:
bq. > /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java, line 192
bq. > <https://reviews.apache.org/r/2909/diff/3/?file=59838#file59838line192>
bq. >
bq. > why are we not reading the system library path for jars. There could be some system jars ,that are not any product specific (like pig, hive) but are commonly used. Also the question of backward compatibility is there.
Is this a real scenario for you or hypothetical? If the latter then I would not worry as users can still use user set library paths.
bq. On 2011-11-23 03:31:31, Mohammad Islam wrote:
bq. > /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java, line 30
bq. > <https://reviews.apache.org/r/2909/diff/3/?file=59834#file59834line30>
bq. >
bq. > But I don't see any other code changes in this file.
My bad, thought the prev comment was for the JavaActionExecutor class, will remove.
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3459
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156284#comment-13156284 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > Overall, it's an important problem.
bq. >
bq. > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
bq. >
bq. > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.
bq.
bq. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.
bq.
bq. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.
bq.
bq. Alejandro Abdelnur wrote:
bq. I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.
bq.
bq. Thanks.
bq.
bq. Mohammad Islam wrote:
bq. I think the "use_version" support is considered here.
bq. Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2
bq. Please review this patch.
bq.
bq. We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)
bq. https://issues.apache.org/jira/browse/OOZIE-555
bq.
bq.
bq.
bq.
bq. Alejandro Abdelnur wrote:
bq. Mmmh (regarding my lost comments).
bq.
bq. OK, the idea is similar, the main difference with what I'm proposing is that in Virag's patch it requires a new WF schema and that the library paths are configured in oozie-site.xml. (BTW, the commit in GH has spurious changes that make difficult to see what is a change and what is not)
bq.
bq. To achieve similar functionality on top of OOZIE-610 we could:
bq.
bq. * Change the sharelib path to contain action library versions, ie:
bq.
bq. share/lib/streaming/0.20.204.0/
bq. share/lib/pig/0.8.0/
bq. share/lib/pig/0.9.0/
bq.
bq. * Have an oozie-site configuration, per action that defines the stable version of the library, ie:
bq.
bq. oozie.action.stable.library.version.streaming=0.20.204.0
bq. oozie.action.stable.library.version.pig=0.8.0
bq.
bq. * The ActionExecutor getShareLibPostFix() method would have the following logic:
bq.
bq. getShareLibPostFix(...) {
bq.
bq. String version = action.config.get("oozie.action.library.version");
bq. if (version == null) {
bq. version = oozie.config.get("oozie.action.stable.library.version");
bq. }
bq. if (version != null) {
bq. //construct the library path using the specified version
bq. }
bq. }
bq.
bq. This approach does not require changes in the WF schema, instead uses a configuration value in case you want a different version than the stable, if not it just works for existing WF apps.
bq.
bq.
bq.
bq. If we are OK with this approach I'll file a JIRA with it.
bq.
bq.
bq. Mohammad Islam wrote:
bq. In addition to your proposal, we want to give the user the flexibility to choose the version *optionally*. User can specify that in wf.xml using use-version. Basically the effective version will be user-specified version , if it is defined in wf.xml. Otherwise, the default one that you mentioned will be the effective version.
bq.
bq. there is already a JIRA on this:
bq. https://issues.apache.org/jira/browse/OOZIE-555
bq. You can add your comments there.
bq.
bq. Alejandro Abdelnur wrote:
bq. In my proposal, if the user sets the configuration property 'oozie.action.library.version' to any value, that version will be used. If the user does not set it the default stable one will be used.
bq.
bq. I'll put my proposal in OOZIE-555
bq.
bq. Mohammad Islam wrote:
bq. The end-user who writes the workflow.xml should have a way of controlling his version. For example, one user tested his pig script againist pig 0.8 another one has used the latest (0.9). We should allow the both users to support that provide pig 0.8 and pig 0.9 are supported versions. I think it should be defined per-action level.
bq.
bq. Also we had the same design discussion few months back.
bq.
yes, it is per action, the 'oozie.action.library.version' property is in the config section of each action, you can get it set to pig 0.8 in one action and to pig 0.9 in another. I've updated OOZIE-555 accordingly to this discussion. thxs
- Alejandro
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155753#comment-13155753 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
bq. On 2011-11-22 23:26:39, Mohammad Islam wrote:
bq. > Overall, it's an important problem.
bq. >
bq. > In every product, we could use some sub-dirs for future (such as lib/ conf/). In other words, oozie should look into $SHARE_LIB/pig/lib/ for classes/jars.
bq. >
bq. > Alternatively, we can coordinate this with multi-versions. There could be multiple directories like $SHARELIB/pig/default/{lib, conf}, $SHARELIB/pig/0.7/{lib, conf}, $SHARELIB/pig/0.8/{lib, conf} etc. I'm leaning towards this way to make it consistent with multi-version support.
bq. >
bq. >
bq.
bq. Alejandro Abdelnur wrote:
bq. Yes, could have lib/ & conf/. However this is not a requirement at the moment, and the change would be transparent when we need it. Like moving from a single lib/ to a per-action one, users apps won't be affected.
bq.
bq. Adding multi version support is a good idea. It would also require changes in the action XML to specify the version.
bq.
bq. Both could be future enhancements, but we should think more the use cases and requirements. The current patch addresses an immediate need that could be augmented later.
bq.
bq. Alejandro Abdelnur wrote:
bq. I'm not against that, actually I like it. But I do that as a follow up feature. Note that, similar from the current single lib/ to multiple dirs (one per action) we are not breaking applications, it is transparent, the case will be when we ad lib/ and conf/ and versions. BTW, handling version requires some thought on how the user will specify the version he/she wants to use (in the schema URI, a config property, a new element; this has to be thought). Because of this I'd like to do things incrementally, the current patch now and a follow up JIRA to discuss your proposal.
bq.
bq. Thanks.
I think the "use_version" support is considered here.
Long time back, Virag had a pending patch : https://github.com/virag/oozie/commit/0f2c7616a4c8d0f669a3df9ca22edb071744a474#diff-2
Please review this patch.
We had a long discussion on this design. Main proposal is captured here (some of your comments lost in the old JIRA system)
https://issues.apache.org/jira/browse/OOZIE-555
- Mohammad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3446
-----------------------------------------------------------
On 2011-11-22 21:03:19, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-22 21:03:19)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205165
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205165
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205165
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205165
bq. /trunk/examples/pom.xml 1205165
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/demo/job.properties 1205165
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205165
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205165
bq. /trunk/examples/src/main/apps/pig/job.properties 1205165
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205165
bq. /trunk/pom.xml 1205165
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205165
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205165
bq. /trunk/src/main/assemblies/examples.xml 1205165
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205165
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156316#comment-13156316 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/#review3490
-----------------------------------------------------------
Ship it!
+1
- Mohammad
On 2011-11-23 21:45:14, Alejandro Abdelnur wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2909/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-11-23 21:45:14)
bq.
bq.
bq. Review request for oozie.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. breaks the Oozie sharelib into one subdir per action type
bq.
bq.
bq. This addresses bug OOZIE-610.
bq. https://issues.apache.org/jira/browse/OOZIE-610
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /trunk/core/pom.xml 1205616
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1205616
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1205616
bq. /trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1205616
bq. /trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1205616
bq. /trunk/docs/src/site/twiki/DG_Examples.twiki 1205616
bq. /trunk/docs/src/site/twiki/DG_QuickStart.twiki 1205616
bq. /trunk/examples/pom.xml 1205616
bq. /trunk/examples/src/main/apps/custom-main/job.properties 1205616
bq. /trunk/examples/src/main/apps/demo/job.properties 1205616
bq. /trunk/examples/src/main/apps/hadoop-el/job.properties 1205616
bq. /trunk/examples/src/main/apps/java-main/job.properties 1205616
bq. /trunk/examples/src/main/apps/pig/job.properties 1205616
bq. /trunk/examples/src/main/apps/streaming/job.properties 1205616
bq. /trunk/pom.xml 1205616
bq. /trunk/sharelib/pig/pom.xml PRE-CREATION
bq. /trunk/sharelib/pom.xml 1205616
bq. /trunk/sharelib/streaming/pom.xml PRE-CREATION
bq. /trunk/src/main/assemblies/distro.xml 1205616
bq. /trunk/src/main/assemblies/examples.xml 1205616
bq. /trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
bq. /trunk/src/main/assemblies/sharelib.xml 1205616
bq.
bq. Diff: https://reviews.apache.org/r/2909/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. built, installed and run pig and streaming examples
bq.
bq.
bq. Thanks,
bq.
bq. Alejandro
bq.
bq.
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154938#comment-13154938 ]
jiraposter@reviews.apache.org commented on OOZIE-610:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2909/
-----------------------------------------------------------
Review request for oozie.
Summary
-------
breaks the Oozie sharelib into one subdir per action type
This addresses bug OOZIE-610.
https://issues.apache.org/jira/browse/OOZIE-610
Diffs
-----
/trunk/core/pom.xml 1204834
/trunk/core/src/main/java/org/apache/oozie/action/ActionExecutor.java 1204834
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1204834
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1204834
/trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1204834
/trunk/core/src/main/java/org/apache/oozie/service/WorkflowAppService.java 1204834
/trunk/docs/src/site/twiki/DG_Examples.twiki 1204834
/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1204834
/trunk/examples/pom.xml 1204834
/trunk/examples/src/main/apps/custom-main/job.properties 1204834
/trunk/examples/src/main/apps/demo/job.properties 1204834
/trunk/examples/src/main/apps/hadoop-el/job.properties 1204834
/trunk/examples/src/main/apps/java-main/job.properties 1204834
/trunk/examples/src/main/apps/pig/job.properties 1204834
/trunk/examples/src/main/apps/streaming/job.properties 1204834
/trunk/pom.xml 1204834
/trunk/sharelib/pig/pom.xml PRE-CREATION
/trunk/sharelib/pom.xml 1204834
/trunk/sharelib/streaming/pom.xml PRE-CREATION
/trunk/src/main/assemblies/distro.xml 1204834
/trunk/src/main/assemblies/examples.xml 1204834
/trunk/src/main/assemblies/partial-sharelib.xml PRE-CREATION
/trunk/src/main/assemblies/sharelib.xml 1204834
Diff: https://reviews.apache.org/r/2909/diff
Testing
-------
built, installed and run pig and streaming examples
Thanks,
Alejandro
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (OOZIE-610) Oozie system share lib should have
jars per action type
Posted by "Alejandro Abdelnur (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OOZIE-610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alejandro Abdelnur resolved OOZIE-610.
--------------------------------------
Resolution: Fixed
> Oozie system share lib should have jars per action type
> -------------------------------------------------------
>
> Key: OOZIE-610
> URL: https://issues.apache.org/jira/browse/OOZIE-610
> Project: Oozie
> Issue Type: Improvement
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
>
> Currently Oozie share lib is a single directory with the the JARs required by the different action types (mapreduce-streaming & pig).
> As more action types are added to Oozie (ie Hive & Sqoop), the Oozie share lib will grow significantly in the number of JARs it has.
> This creates a few issues:
> * The classpath for an action grows significantly
> * For a given action only a portion of the classpath is useful, the rest is dead code
> * As more actions are added, the chances of conflicting dependencies grows.
> As I'm working on integrating Hive action (OOZIE-68), I'm running into the issue described in the #3 bullet item. Pig 0.9.0 requires antlr-runtime 3.4 to work properly, while Hive requires antlr-runtime 3.0.1 to work properly.
> Because the current sharelib aggregates all dependencies and resolves into a single version of each one of them, only one version of antlr-runtime makes it. And because of this, either Pig or Hive works but not both.
> This JIRA proposes to add one subdirectory per action (when the action requires specific JARs) to the Oozie share lib, for example:
> share/lib/pig/*
> share/lib/streaming/*
> share/lib/hive/*
> share/lib/sqoop/*
> Then, the ActionExecutor for each action type will add to the action classpath all the JARs for the corresponding action only.
> This would move the resolution for the Oozie share lib from the submit-command to the action-executor.
> Note that this change will not break workflow applications using Oozie system share library as the change will be transparent to applications.
> Finally, the sharelib maven submodule becomes an aggregator for the sharelibs for each action.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira