You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Purshotam Shah <pu...@yahoo-inc.com> on 2013/10/18 19:22:51 UTC
Review Request 14741: Setup sharelib using script and pickup latest(honor
ship.launcher) and remove DFS dependency at startup.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
Review request for oozie.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Oct. 18, 2013, 6:30 p.m., Robert Kanter wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 371
> > <https://reviews.apache.org/r/14741/diff/2/?file=366370#file366370line371>
> >
> > Please double check, but I'm pretty sure that you can reuse the same SimpleDateFormat object instead of creating a new one each time. So instead of this method that creates a new one with the same pattern, you can just use this:
> >
> > private static final SimpleDateFormat dateFormat = new SimpleDateFormat("yyyyMMddHHmmss");
Agree that defining static instance is better option.
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27197
-----------------------------------------------------------
On Oct. 21, 2013, 4:01 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Oct. 21, 2013, 4:01 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
>
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
>
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Robert Kanter <rk...@cloudera.com>.
> On Oct. 18, 2013, 6:30 p.m., Robert Kanter wrote:
> >
Can you also update the documentation?
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27197
-----------------------------------------------------------
On Oct. 18, 2013, 5:22 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Oct. 18, 2013, 5:22 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
>
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
>
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27197
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment52900>
Typo: getPathRecursively
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment52901>
"doesn't exist"
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment52902>
Can you use curly braces for the if statement even though its one line? It's much less error prone in the future if we add more lines or change something.
i.e.
if (...) {
purgeLibs(...);
}
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment52903>
Please double check, but I'm pretty sure that you can reuse the same SimpleDateFormat object instead of creating a new one each time. So instead of this method that creates a new one with the same pattern, you can just use this:
private static final SimpleDateFormat dateFormat = new SimpleDateFormat("yyyyMMddHHmmss");
- Robert Kanter
On Oct. 18, 2013, 5:22 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Oct. 18, 2013, 5:22 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
>
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
>
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Nov. 8, 2013, 10:07 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 534
> > <https://reviews.apache.org/r/14741/diff/8/?file=380812#file380812line534>
> >
> > What is the type for?
Removed
> On Nov. 8, 2013, 10:07 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 125
> > <https://reviews.apache.org/r/14741/diff/8/?file=380812#file380812line125>
> >
> > Pass the launcherlibPath
Function is called from multiple places.
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28558
-----------------------------------------------------------
On Nov. 8, 2013, 10:50 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2013, 10:50 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28558
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment55358>
You can just do this with systemLibPath and appPath outside of the for loop
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55378>
"oozie.". Code needs to be formatted
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55360>
isShipLauncherEnabled or isShipLauncher
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55362>
Pass the launcherlibPath
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55430>
log statement here
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55368>
Rename to getSystemLibJars
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55369>
loadShareLibMetaFile
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55429>
log statement here
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55376>
Create a new fs based on the file uri. That way it will work even if the file is on local filesystem or hdfs or webhdfs
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55373>
Add a log statement here as well as to what is being added to sharelib
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55374>
What is the type for?
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/14741/#comment55379>
Add an example
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java
<https://reviews.apache.org/r/14741/#comment55377>
close the stream
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment55431>
space after full stop.
* key will be the sharelib name for the action
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment55433>
a comma separated list of DFS directories or jar files.
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
<https://reviews.apache.org/r/14741/#comment55380>
create a new timestamped version of oozie sharelib
- Rohini Palaniswamy
On Nov. 8, 2013, 6:33 a.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2013, 6:33 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28680
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment55620>
Also need to move this to AG_Install doc instead of quick start.
- Rohini Palaniswamy
On Nov. 8, 2013, 10:50 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2013, 10:50 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Nov. 11, 2013, 7:28 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 473
> > <https://reviews.apache.org/r/14741/diff/9/?file=381242#file381242line473>
> >
> > ??
Sharelib load attempt will only once.
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28676
-----------------------------------------------------------
On Nov. 12, 2013, 7:56 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2013, 7:56 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28676
-----------------------------------------------------------
Just few minor comments. Javadoc needs to be updated in places based on review comments addressed.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55601>
Use caps for static variable
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55596>
Do not add non-Javadoc comments
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55606>
Needs updating
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55607>
To be removed
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55605>
Needs updating
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55604>
To be removed
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55600>
shareLibMap - need to rename in few other places too. Left over from old patch.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55599>
??
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55603>
To be removed
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55602>
To be removed
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/14741/#comment55608>
Keep example consistent with open source pig and hadoop. Also use hdfs:/// path prefix
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment55609>
Example here as well
- Rohini Palaniswamy
On Nov. 8, 2013, 10:50 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 8, 2013, 10:50 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Nov. 12, 2013, 8:19 p.m., Robert Kanter wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 289
> > <https://reviews.apache.org/r/14741/diff/9/?file=381242#file381242line289>
> >
> > Instead of using a synchronized block here, can you use the Locks service? This way, when Oozie HA is enabled, it will automatically use the distributed locks so multiple Oozie servers won't try to update the sharelib/launcherlib at the same time. You can use a lock named "org.apache.oozie.service.ShareLibService".
> >
> > We should also put this below where the other synchronized block is.
> >
> > There are probably other places where this could be a problem too, like purging the sharelib. I think that simply surrounding it with a lock from the Locks service should take care of these cases.
> >
> > Also, I assume we only want one server to deal with purging the sharelib, right? If so, you can use the #isFirstServer() method from the JobsConcurrencyService to check if the server should be doing the purge. It always returns true for non-HA and only returns true for one server with HA.
synchronized is used for caching file from DFS. There is no update here. It used in scenario where caching of sharelib at startup failed. At the first job submission we try to cache sharelib. We synchronized the block with double check to take care of multiple job submission at same time.
We might have to think for system launcher jar. When SHIP_LAUNCHER_JAR=true server tries to copy jars from local to DFS. I believe copy of system launcher jar should be ok. With current implementation system launcher jar are always copied to DFS with new timestamp directory. Each server will be using his own version of system share lib.
We might have to think of purging, currently server delete anything older than 7 days.In case of HA, restart of 1 server might delete system share lib of long running other Oozie server.
Yup, only one server should purge the sharelib, we will use isFirstServer() for that.
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28752
-----------------------------------------------------------
On Nov. 12, 2013, 7:56 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2013, 7:56 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28752
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55751>
Instead of using a synchronized block here, can you use the Locks service? This way, when Oozie HA is enabled, it will automatically use the distributed locks so multiple Oozie servers won't try to update the sharelib/launcherlib at the same time. You can use a lock named "org.apache.oozie.service.ShareLibService".
We should also put this below where the other synchronized block is.
There are probably other places where this could be a problem too, like purging the sharelib. I think that simply surrounding it with a lock from the Locks service should take care of these cases.
Also, I assume we only want one server to deal with purging the sharelib, right? If so, you can use the #isFirstServer() method from the JobsConcurrencyService to check if the server should be doing the purge. It always returns true for non-HA and only returns true for one server with HA.
- Robert Kanter
On Nov. 12, 2013, 7:56 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2013, 7:56 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28839
-----------------------------------------------------------
Ship it!
Ship It!
- Rohini Palaniswamy
On Nov. 13, 2013, 9:57 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2013, 9:57 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 1541609
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Nov. 13, 2013, 9:57 p.m.)
Review request for oozie.
Changes
-------
review comment.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
This can be configured in oozie-site.xml as :
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key is the action key and value is the DFS location of sharelib files.
</description>
</property>
--------------------
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1541609
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1541609
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1541609
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1541609
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1541609
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1541609
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1541609
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 1541609
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28836
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java
<https://reviews.apache.org/r/14741/#comment55852>
Please reuse the method from SharelibService instead of duplicating code
- Rohini Palaniswamy
On Nov. 13, 2013, 9:36 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 13, 2013, 9:36 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1541609
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 1541609
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Nov. 13, 2013, 9:36 p.m.)
Review request for oozie.
Changes
-------
test fix.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
This can be configured in oozie-site.xml as :
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key is the action key and value is the DFS location of sharelib files.
</description>
</property>
--------------------
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1541609
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1541609
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1541609
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1541609
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1541609
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1541609
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1541609
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 1541609
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28770
-----------------------------------------------------------
Ship it!
Ship It!
- Rohini Palaniswamy
On Nov. 12, 2013, 10:41 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2013, 10:41 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Nov. 12, 2013, 10:41 p.m.)
Review request for oozie.
Changes
-------
review comment.
We will address HA issues @https://issues.apache.org/jira/browse/OOZIE-1609.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
This can be configured in oozie-site.xml as :
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key is the action key and value is the DFS location of sharelib files.
</description>
</property>
--------------------
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28761
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55762>
shareLibLoaded = true should be here.
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki
<https://reviews.apache.org/r/14741/#comment55766>
ship a launcher jar to hdfs
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki
<https://reviews.apache.org/r/14741/#comment55765>
Remove this line. It should be faster now as we copy to hdfs only once during startup
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki
<https://reviews.apache.org/r/14741/#comment55763>
sharelib CLI.
directory. While starting, server also purges sharelib directories which are
- Rohini Palaniswamy
On Nov. 12, 2013, 7:56 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 12, 2013, 7:56 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Nov. 12, 2013, 7:56 p.m.)
Review request for oozie.
Changes
-------
Review comment..
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
This can be configured in oozie-site.xml as :
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key is the action key and value is the DFS location of sharelib files.
</description>
</property>
--------------------
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Nov. 8, 2013, 10:50 p.m.)
Review request for oozie.
Changes
-------
Review changes.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
This can be configured in oozie-site.xml as :
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key is the action key and value is the DFS location of sharelib files.
</description>
</property>
--------------------
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Nov. 8, 2013, 6:33 a.m.)
Review request for oozie.
Changes
-------
Following changes after today's meeting
1. Tag file can be in DFS.
2. Tag file can be Hadoop supported conf file(xml, prop).
3. Should purge even if property is not set.
4. Tag file key will be prefix with Oozie.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
This can be configured in oozie-site.xml as :
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key is the action key and value is the DFS location of sharelib files.
</description>
</property>
--------------------
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Nov. 7, 2013, 5:59 a.m.)
Review request for oozie.
Changes
-------
Review changes.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
This can be configured in oozie-site.xml as :
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key is the action key and value is the DFS location of sharelib files.
</description>
</property>
--------------------
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Nov. 5, 2013, 6:54 p.m., Virag Kothari wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 427
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line427>
> >
> > if path is null if all directory's are somehow malformed, then need to be handled by the caller function
This is taken care while adding path to cache.
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28196
-----------------------------------------------------------
On Nov. 7, 2013, 5:59 a.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 7, 2013, 5:59 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Robert Kanter <rk...@cloudera.com>.
> On Nov. 5, 2013, 6:54 p.m., Virag Kothari wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 252
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line252>
> >
> > it might happen that this property was true during previous instance of oozie running but now its false. So need to remove the if condition. Same for the mapping file below.
FYI: when getting a boolean from a Configuration, there is a .getBoolean method you can use instead of having to do Boolean.parseBoolean
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28196
-----------------------------------------------------------
On Nov. 1, 2013, 5:58 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 1, 2013, 5:58 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28196
-----------------------------------------------------------
minor comments
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54860>
instead of reading this value everytime, better read it once (constructor) and store in a variable.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54863>
remove /// at end
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54864>
javadoc
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54872>
it might happen that this property was true during previous instance of oozie running but now its false. So need to remove the if condition. Same for the mapping file below.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54873>
it might happen that this property was true during previous instance of oozie running but now its false. So need to remove the if condition. Same for the mapping file condition below.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54866>
remove the blank line, same comment for other places in the function.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54867>
remove
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54871>
javadoc
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54870>
javadoc
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54869>
javadoc
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54868>
if path is null if all directory's are somehow malformed, then need to be handled by the caller function
- Virag Kothari
On Nov. 1, 2013, 5:58 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 1, 2013, 5:58 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 97
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line97>
> >
> > It does not need launcherlibPath. Needs the system lib path only.
> >
> > Also log a error message if exception in purge. don't fail
exception is handled and logged at parent method.
> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 287
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line287>
> >
> > how does this work without prefix?
This will be taken care by..
PathFilter directoryFilter = new PathFilter() {
@Override
public boolean accept(Path path) {
return path.getName().startsWith(prefix);
}
};
> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 252
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line252>
> >
> > Do purge irrespective of the setting.
If property is not set, then why to delete something and throw exception.
> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 257
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line257>
> >
> > Do purge irrespective of the setting.
If property is not set, then why to delete something and throw exception.
> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 128
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line128>
> >
> > This code should not be removed
Discussed with Viraj, we need this method.
details @ https://issues.apache.org/jira/browse/OOZIE-1528
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28053
-----------------------------------------------------------
On Nov. 7, 2013, 5:59 a.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 7, 2013, 5:59 a.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28053
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54913>
Move creation of fs after this based on systemLibPath and move it out of for loop
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54573>
Use a different method name like getActionShareLibJars
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54909>
Rename to addShippedLauncherShareLibForAction
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54908>
Revert ///
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54948>
Call getActionLauncherJars. return if listOfPaths is null
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54932>
rename to launcherJarsMap and have a separate shareLibMap
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54850>
Construct the tmp path inside the if block.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54933>
Extract this out to a separate method shipLauncherJars
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54848>
Make this as a local variable instead of parsing every time
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54849>
It does not need launcherlibPath. Needs the system lib path only.
Also log a error message if exception in purge. don't fail
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54915>
Should not be needing this on init.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54925>
Combine into one log statement.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54917>
launcherLibPath
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54874>
This code should not be removed
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54919>
Not required
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54927>
Rename method to getActionLauncherJars
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54947>
if shipLauncher setting is true &&
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54949>
copyLauncherJars
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54946>
throw the exception. do not log.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54929>
inline this method in init() and remove this method.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54922>
Do purge irrespective of the setting.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54923>
Do purge irrespective of the setting.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54924>
launcherJarLibPath is not required in method signature
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54875>
how does this work without prefix?
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54934>
Method can be removed
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54935>
Call method as cacheShareLibMapping
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54936>
Use a different map for this - shareLibMap
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54938>
Why is type required in method signature. Can be removed
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54944>
properties
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54943>
PropertiesFile
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54942>
Can remove this method
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54940>
LauncherLibPath
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54941>
tmpLauncherLibPath
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54937>
getLatestLibPath
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54876>
have this as a local variable
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment54877>
Remove upgrade
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment54878>
remove "new or upgrading existing"
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment54879>
Specify the format lib_<timestamp>
timestamped instead of time-stamp
- Rohini Palaniswamy
On Nov. 1, 2013, 5:58 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 1, 2013, 5:58 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Nov. 1, 2013, 5:58 p.m.)
Review request for oozie.
Changes
-------
Review changes..
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
This can be configured in oozie-site.xml as :
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key is the action key and value is the DFS location of sharelib files.
</description>
</property>
--------------------
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
> On Oct. 29, 2013, 8:11 p.m., Virag Kothari wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 478
> > <https://reviews.apache.org/r/14741/diff/5/?file=372236#file372236line478>
> >
> > if listOfpaths is null, then we need to have previous logic of reading the files from hdfs and adding to cache.
Sharelib is empty, trying to repopulate them.
- Purshotam
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27715
-----------------------------------------------------------
On Nov. 1, 2013, 5:58 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Nov. 1, 2013, 5:58 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27715
-----------------------------------------------------------
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/conf/oozie-site.xml
<https://reviews.apache.org/r/14741/#comment53834>
uncomment and have it in oozie-default
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment53804>
if listOfpaths is null, then we need to have previous logic of reading the files from hdfs and adding to cache.
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment53805>
lowercase 'o' in Oozie
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment53808>
call if property file doesn't exist
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment53812>
remove if not used
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
<https://reviews.apache.org/r/14741/#comment53836>
create a new timestamp dir for upgrade
- Virag Kothari
On Oct. 28, 2013, 5:58 p.m., Purshotam Shah wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
>
> (Updated Oct. 28, 2013, 5:58 p.m.)
>
>
> Review request for oozie.
>
>
> Bugs: OOZIE-1584
> https://issues.apache.org/jira/browse/OOZIE-1584
>
>
> Repository: oozie
>
>
> Description
> -------
>
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
> server start.
>
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
>
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
>
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
> This can be configured in oozie-site.xml as :
> <property>
> <name>oozie.service.ShareLibService.mapping.file</name>
> <value></value>
> <description>
> Sharelib mapping files contains list of key=value,
> where key is the action key and value is the DFS location of sharelib files.
> </description>
> </property>
>
> --------------------
> DFS after generating launcher jar by ooozie server and sharelib by cli
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
>
>
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property) is purged.
> We always keep 2 set of directory, purging happens if number of sharelib
> directory > 2.
>
>
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
>
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
>
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/conf/oozie-site.xml 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
> http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
>
> Diff: https://reviews.apache.org/r/14741/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Purshotam Shah
>
>
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Oct. 28, 2013, 5:58 p.m.)
Review request for oozie.
Changes
-------
+ Sharelib mapping file support.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description (updated)
-------
Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
This can be configured in oozie-site.xml as :
<property>
<name>oozie.service.ShareLibService.mapping.file</name>
<value></value>
<description>
Sharelib mapping files contains list of key=value,
where key is the action key and value is the DFS location of sharelib files.
</description>
</property>
--------------------
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/conf/oozie-site.xml 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Oct. 21, 2013, 10:09 p.m.)
Review request for oozie.
Changes
-------
Review changes + Some more test-cases.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah
Re: Review Request 14741: Setup sharelib using script and pickup
latest(honor ship.launcher) and remove DFS dependency at startup.
Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------
(Updated Oct. 21, 2013, 4:01 p.m.)
Review request for oozie.
Changes
-------
with review changes.
Bugs: OOZIE-1584
https://issues.apache.org/jira/browse/OOZIE-1584
Repository: oozie
Description
-------
Supported features.
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at
server start.
2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.
3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).
DFS after generating launcher jar by ooozie server and sharelib by cli
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$
Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property) is purged.
We always keep 2 set of directory, purging happens if number of sharelib
directory > 2.
Multiple version.
-------------------
To support multiple version, sharelib will be in below format.
Sharelib will cache all versions and return list of dfs files for a particular
version.
makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$
Diffs (updated)
-----
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127
http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127
Diff: https://reviews.apache.org/r/14741/diff/
Testing
-------
Thanks,
Purshotam Shah