You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Virag Kothari <vi...@yahoo-inc.com> on 2013/09/02 08:57:01 UTC
Review Request 13931: OOZIE-1461: provide an option to auto-deploy launcher
jar onto HDFS system libpath
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13931/
-----------------------------------------------------------
Review request for oozie.
Bugs: OOZIE-1461
https://issues.apache.org/jira/browse/OOZIE-1461
Repository: oozie
Description
-------
Added sharelib service which uploads all sharelib jars containing launcher classes to hdfs
JavaActionExecutor will add this to DC.
Sharelib service will always keep minimum 2 recent version of sharelib. Other sharelibs older than 7 days (configurable) will be deleted.
Diffs
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/service/ActionService.java 1519337
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java PRE-CREATION
trunk/core/src/main/resources/oozie-default.xml 1519337
trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 1519337
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1519337
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1519337
trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1519337
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1519337
trunk/sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1519337
trunk/sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 1519337
trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1519337
trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java 1519337
Diff: https://reviews.apache.org/r/13931/diff/
Testing
-------
Unit test case and ran pig, mapreduce examples
Thanks,
Virag Kothari
Re: Review Request 13931: OOZIE-1461: provide an option to auto-deploy
launcher jar onto HDFS system libpath
Posted by Virag Kothari <vi...@yahoo-inc.com>.
> On Sept. 6, 2013, 2:16 a.m., Mona Chitnis wrote:
> > Does this mean the action lib jars also lie around in Oozie-server WEB-INF lib? Considering that is < 10 jars, that should be fine.
They were already in WEB-INF when sharelib refactoring was done.
> On Sept. 6, 2013, 2:16 a.m., Mona Chitnis wrote:
> > trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 577
> > <https://reviews.apache.org/r/13931/diff/4/?file=348235#file348235line577>
> >
> > Can we add a debug level log statement here for all the JARs added
This are the system level jars added by Sharelib service where we already have the log statements. So, doesn't seem to add much value as they will be always be the same.
> On Sept. 6, 2013, 2:16 a.m., Mona Chitnis wrote:
> > trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java, line 51
> > <https://reviews.apache.org/r/13931/diff/4/?file=348237#file348237line51>
> >
> > So we don't need this class OoziePigStats to be added anymore?
we do findContainingJar in sharelibservice, so the jar which has ooziePigstats will be picked up as it also has the PigMain class
> On Sept. 6, 2013, 2:16 a.m., Mona Chitnis wrote:
> > trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 186
> > <https://reviews.apache.org/r/13931/diff/4/?file=348242#file348242line186>
> >
> > How is "2" determined? Shouldn't we purge based on LRU?
we first sort in desc order based on timestamp and then keep the latest two, so kind of LRU only
- Virag
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13931/#review25951
-----------------------------------------------------------
On Sept. 5, 2013, 5:47 p.m., Virag Kothari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13931/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2013, 5:47 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1461
> https://issues.apache.org/jira/browse/OOZIE-1461
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Added sharelib service which uploads all sharelib jars containing launcher classes to hdfs
> JavaActionExecutor will add this to DC.
> Sharelib service will always keep minimum 2 recent version of sharelib. Other sharelibs older than 7 days (configurable) will be deleted.
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/service/ActionService.java 1520378
> trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java PRE-CREATION
> trunk/core/src/main/resources/oozie-default.xml 1520378
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 1520378
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1520378
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1520378
> trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java PRE-CREATION
> trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1520378
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1520378
> trunk/sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java 1520378
> trunk/sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1520378
> trunk/sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 1520378
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1520378
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java 1520378
>
> Diff: https://reviews.apache.org/r/13931/diff/
>
>
> Testing
> -------
>
> Unit test case and ran pig, mapreduce examples
>
>
> Thanks,
>
> Virag Kothari
>
>
Re: Review Request 13931: OOZIE-1461: provide an option to auto-deploy
launcher jar onto HDFS system libpath
Posted by Mona Chitnis <mo...@yahoo.in>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13931/#review25951
-----------------------------------------------------------
Does this mean the action lib jars also lie around in Oozie-server WEB-INF lib? Considering that is < 10 jars, that should be fine.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50656>
Can we add a debug level log statement here for all the JARs added
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50657>
So we don't need this class OoziePigStats to be added anymore?
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50658>
How is "2" determined? Shouldn't we purge based on LRU?
- Mona Chitnis
On Sept. 5, 2013, 5:47 p.m., Virag Kothari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13931/
> -----------------------------------------------------------
>
> (Updated Sept. 5, 2013, 5:47 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1461
> https://issues.apache.org/jira/browse/OOZIE-1461
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Added sharelib service which uploads all sharelib jars containing launcher classes to hdfs
> JavaActionExecutor will add this to DC.
> Sharelib service will always keep minimum 2 recent version of sharelib. Other sharelibs older than 7 days (configurable) will be deleted.
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1520378
> trunk/core/src/main/java/org/apache/oozie/service/ActionService.java 1520378
> trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java PRE-CREATION
> trunk/core/src/main/resources/oozie-default.xml 1520378
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 1520378
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1520378
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1520378
> trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java PRE-CREATION
> trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1520378
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1520378
> trunk/sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java 1520378
> trunk/sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1520378
> trunk/sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 1520378
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1520378
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java 1520378
>
> Diff: https://reviews.apache.org/r/13931/diff/
>
>
> Testing
> -------
>
> Unit test case and ran pig, mapreduce examples
>
>
> Thanks,
>
> Virag Kothari
>
>
Re: Review Request 13931: OOZIE-1461: provide an option to auto-deploy
launcher jar onto HDFS system libpath
Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13931/
-----------------------------------------------------------
(Updated Sept. 5, 2013, 5:47 p.m.)
Review request for oozie.
Changes
-------
addressed rohini's comments
Bugs: OOZIE-1461
https://issues.apache.org/jira/browse/OOZIE-1461
Repository: oozie
Description
-------
Added sharelib service which uploads all sharelib jars containing launcher classes to hdfs
JavaActionExecutor will add this to DC.
Sharelib service will always keep minimum 2 recent version of sharelib. Other sharelibs older than 7 days (configurable) will be deleted.
Diffs (updated)
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1520378
trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1520378
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1520378
trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1520378
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1520378
trunk/core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java 1520378
trunk/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 1520378
trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1520378
trunk/core/src/main/java/org/apache/oozie/service/ActionService.java 1520378
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java PRE-CREATION
trunk/core/src/main/resources/oozie-default.xml 1520378
trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 1520378
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1520378
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1520378
trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1520378
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1520378
trunk/sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java 1520378
trunk/sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1520378
trunk/sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 1520378
trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1520378
trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java 1520378
Diff: https://reviews.apache.org/r/13931/diff/
Testing
-------
Unit test case and ran pig, mapreduce examples
Thanks,
Virag Kothari
Re: Review Request 13931: OOZIE-1461: provide an option to auto-deploy
launcher jar onto HDFS system libpath
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13931/#review25911
-----------------------------------------------------------
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50559>
addToCache does lot of checks which is not required. Why not directly add to DistributedCache and create symlink?
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50564>
tmpShareLibPath
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50567>
copyLauncherJarsToShareLib(fs, tmpShareLibPath)
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50563>
Don't need this line
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50570>
(1000 * 60 * 60 * 24 * conf.getInt(LAUNCHERJAR_LIB_RETENTION, 7))) outside the loop
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50572>
Nitpick. Comment on the English Language (Not java language :) )
"Deleted". Capitalize first letter in the sentence.
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50573>
clear the maps in destroy
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java
<https://reviews.apache.org/r/13931/#comment50578>
as maven has target/classes in classpath and not the jar because test phase is before package phase
- Rohini Palaniswamy
On Sept. 4, 2013, 10:02 p.m., Virag Kothari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13931/
> -----------------------------------------------------------
>
> (Updated Sept. 4, 2013, 10:02 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1461
> https://issues.apache.org/jira/browse/OOZIE-1461
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Added sharelib service which uploads all sharelib jars containing launcher classes to hdfs
> JavaActionExecutor will add this to DC.
> Sharelib service will always keep minimum 2 recent version of sharelib. Other sharelibs older than 7 days (configurable) will be deleted.
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/service/ActionService.java 1519337
> trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java PRE-CREATION
> trunk/core/src/main/resources/oozie-default.xml 1519337
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 1519337
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1519337
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1519337
> trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java PRE-CREATION
> trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1519337
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1519337
> trunk/sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java 1519337
> trunk/sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1519337
> trunk/sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 1519337
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1519337
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java 1519337
>
> Diff: https://reviews.apache.org/r/13931/diff/
>
>
> Testing
> -------
>
> Unit test case and ran pig, mapreduce examples
>
>
> Thanks,
>
> Virag Kothari
>
>
Re: Review Request 13931: OOZIE-1461: provide an option to auto-deploy
launcher jar onto HDFS system libpath
Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13931/
-----------------------------------------------------------
(Updated Sept. 4, 2013, 10:02 p.m.)
Review request for oozie.
Changes
-------
Updated based on Rohini's comments
Bugs: OOZIE-1461
https://issues.apache.org/jira/browse/OOZIE-1461
Repository: oozie
Description
-------
Added sharelib service which uploads all sharelib jars containing launcher classes to hdfs
JavaActionExecutor will add this to DC.
Sharelib service will always keep minimum 2 recent version of sharelib. Other sharelibs older than 7 days (configurable) will be deleted.
Diffs (updated)
-----
trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1519337
trunk/core/src/main/java/org/apache/oozie/service/ActionService.java 1519337
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java PRE-CREATION
trunk/core/src/main/resources/oozie-default.xml 1519337
trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 1519337
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1519337
trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1519337
trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java PRE-CREATION
trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1519337
trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1519337
trunk/sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java 1519337
trunk/sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1519337
trunk/sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 1519337
trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1519337
trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java 1519337
Diff: https://reviews.apache.org/r/13931/diff/
Testing
-------
Unit test case and ran pig, mapreduce examples
Thanks,
Virag Kothari
Re: Review Request 13931: OOZIE-1461: provide an option to auto-deploy
launcher jar onto HDFS system libpath
Posted by Virag Kothari <vi...@yahoo-inc.com>.
> On Sept. 4, 2013, 6:36 p.m., Rohini Palaniswamy wrote:
> > trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 624
> > <https://reviews.apache.org/r/13931/diff/2/?file=347279#file347279line624>
> >
> > Why public?
Need for test case in TestShareLibService
> On Sept. 4, 2013, 6:36 p.m., Rohini Palaniswamy wrote:
> > trunk/core/src/main/resources/oozie-default.xml, line 1621
> > <https://reviews.apache.org/r/13931/diff/2/?file=347287#file347287line1621>
> >
> > Do we need hdfs:/// ?
yes, ryota found that it was required.
> On Sept. 4, 2013, 6:36 p.m., Rohini Palaniswamy wrote:
> > trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java, line 76
> > <https://reviews.apache.org/r/13931/diff/2/?file=347288#file347288line76>
> >
> > Why public?
Need for test case in TestShareLibService
- Virag
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13931/#review25851
-----------------------------------------------------------
On Sept. 4, 2013, 10:02 p.m., Virag Kothari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13931/
> -----------------------------------------------------------
>
> (Updated Sept. 4, 2013, 10:02 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1461
> https://issues.apache.org/jira/browse/OOZIE-1461
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Added sharelib service which uploads all sharelib jars containing launcher classes to hdfs
> JavaActionExecutor will add this to DC.
> Sharelib service will always keep minimum 2 recent version of sharelib. Other sharelibs older than 7 days (configurable) will be deleted.
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/service/ActionService.java 1519337
> trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java PRE-CREATION
> trunk/core/src/main/resources/oozie-default.xml 1519337
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 1519337
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1519337
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1519337
> trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java PRE-CREATION
> trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1519337
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1519337
> trunk/sharelib/hive/src/test/java/org/apache/oozie/action/hadoop/TestHiveActionExecutor.java 1519337
> trunk/sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1519337
> trunk/sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 1519337
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1519337
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java 1519337
>
> Diff: https://reviews.apache.org/r/13931/diff/
>
>
> Testing
> -------
>
> Unit test case and ran pig, mapreduce examples
>
>
> Thanks,
>
> Virag Kothari
>
>
Re: Review Request 13931: OOZIE-1461: provide an option to auto-deploy
launcher jar onto HDFS system libpath
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/13931/#review25851
-----------------------------------------------------------
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50413>
Should still leave this as COMMON
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50521>
Have a static getCommonLauncherClasses method and make it return the common classes.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50415>
Instead of this retain
classes.addAll(Services.get().get(URIHandlerService.class).getClassesForLauncher());
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50473>
Should not be
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50491>
getActionSystemLibCommonJars - Make this method return a string (variable at class level. needs to be loaded only once) of the comma separated list of jars.
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50494>
You can also get the filesystem from sharedlibservice
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50492>
getActionSystemLibDir - You can also make this return a cached string
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50474>
Why public?
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50496>
Add common sharelibs for Oozie and launcher jars
trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/13931/#comment50497>
addSystemShareLibForAction
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50499>
ShareLibService.temp.sharelib.retention.days
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50501>
Filesystem as local variable
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50505>
Do a set of strings to eliminate duplicates
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50478>
Do we need a set permission after this?
trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/13931/#comment50477>
IOException
trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/13931/#comment50479>
Do we need hdfs:/// ?
trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java
<https://reviews.apache.org/r/13931/#comment50480>
Why public?
- Rohini Palaniswamy
On Sept. 2, 2013, 6:56 a.m., Virag Kothari wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/13931/
> -----------------------------------------------------------
>
> (Updated Sept. 2, 2013, 6:56 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1461
> https://issues.apache.org/jira/browse/OOZIE-1461
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Added sharelib service which uploads all sharelib jars containing launcher classes to hdfs
> JavaActionExecutor will add this to DC.
> Sharelib service will always keep minimum 2 recent version of sharelib. Other sharelibs older than 7 days (configurable) will be deleted.
>
>
> Diffs
> -----
>
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/DistcpActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/HiveActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/MapReduceActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/PigActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ScriptLanguageActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/ShellActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/action/hadoop/SqoopActionExecutor.java 1519337
> trunk/core/src/main/java/org/apache/oozie/service/ActionService.java 1519337
> trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java PRE-CREATION
> trunk/core/src/main/resources/oozie-default.xml 1519337
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/ActionExecutorTestCase.java 1519337
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestJavaActionExecutor.java 1519337
> trunk/core/src/test/java/org/apache/oozie/action/hadoop/TestShellActionExecutor.java 1519337
> trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java PRE-CREATION
> trunk/core/src/test/java/org/apache/oozie/test/XFsTestCase.java 1519337
> trunk/core/src/test/java/org/apache/oozie/test/XTestCase.java 1519337
> trunk/sharelib/pig/src/test/java/org/apache/oozie/action/hadoop/TestPigActionExecutor.java 1519337
> trunk/sharelib/sqoop/src/test/java/org/apache/oozie/action/hadoop/TestSqoopActionExecutor.java 1519337
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutor.java 1519337
> trunk/sharelib/streaming/src/test/java/org/apache/oozie/action/hadoop/TestMapReduceActionExecutorUberJar.java 1519337
>
> Diff: https://reviews.apache.org/r/13931/diff/
>
>
> Testing
> -------
>
> Unit test case and ran pig, mapreduce examples
>
>
> Thanks,
>
> Virag Kothari
>
>