You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oozie.apache.org by Purshotam Shah <pu...@yahoo-inc.com> on 2013/10/18 19:22:51 UTC

Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

Review request for oozie.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.

1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).


DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.

> On Oct. 18, 2013, 6:30 p.m., Robert Kanter wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 371
> > <https://reviews.apache.org/r/14741/diff/2/?file=366370#file366370line371>
> >
> >     Please double check, but I'm pretty sure that you can reuse the same SimpleDateFormat object instead of creating a new one each time.  So instead of this method that creates a new one with the same pattern, you can just use this:
> >     
> >     private static final SimpleDateFormat dateFormat = new SimpleDateFormat("yyyyMMddHHmmss");

Agree that defining static instance is better option.


- Purshotam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27197
-----------------------------------------------------------


On Oct. 21, 2013, 4:01 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Oct. 21, 2013, 4:01 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> 
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Robert Kanter <rk...@cloudera.com>.

> On Oct. 18, 2013, 6:30 p.m., Robert Kanter wrote:
> >

Can you also update the documentation?


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27197
-----------------------------------------------------------


On Oct. 18, 2013, 5:22 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Oct. 18, 2013, 5:22 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> 
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27197
-----------------------------------------------------------



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment52900>

    Typo: getPathRecursively



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment52901>

    "doesn't exist"



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment52902>

    Can you use curly braces for the if statement even though its one line?  It's much less error prone in the future if we add more lines or change something.
    
    i.e.
    
    if (...) {
       purgeLibs(...);
    }



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment52903>

    Please double check, but I'm pretty sure that you can reuse the same SimpleDateFormat object instead of creating a new one each time.  So instead of this method that creates a new one with the same pattern, you can just use this:
    
    private static final SimpleDateFormat dateFormat = new SimpleDateFormat("yyyyMMddHHmmss");


- Robert Kanter


On Oct. 18, 2013, 5:22 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Oct. 18, 2013, 5:22 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> 
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.

> On Nov. 8, 2013, 10:07 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 534
> > <https://reviews.apache.org/r/14741/diff/8/?file=380812#file380812line534>
> >
> >     What is the type for?

Removed


> On Nov. 8, 2013, 10:07 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 125
> > <https://reviews.apache.org/r/14741/diff/8/?file=380812#file380812line125>
> >
> >     Pass the launcherlibPath

Function is called from multiple places.


- Purshotam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28558
-----------------------------------------------------------


On Nov. 8, 2013, 10:50 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 8, 2013, 10:50 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28558
-----------------------------------------------------------



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment55358>

    You can just do this with systemLibPath and appPath outside of the for loop



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55378>

    "oozie.". Code needs to be formatted



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55360>

    isShipLauncherEnabled or isShipLauncher



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55362>

    Pass the launcherlibPath



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55430>

    log statement here



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55368>

    Rename to getSystemLibJars



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55369>

    loadShareLibMetaFile



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55429>

    log statement here



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55376>

    Create a new fs based on the file uri. That way it will work even if the file is on local filesystem or hdfs or webhdfs



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55373>

    Add a log statement here as well as to what is being added to sharelib



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55374>

    What is the type for?



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/14741/#comment55379>

    Add an example



http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java
<https://reviews.apache.org/r/14741/#comment55377>

    close the stream



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment55431>

    space after full stop.
    
    * key will be the sharelib name for the action



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment55433>

    a comma separated list of DFS directories or jar files.



http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
<https://reviews.apache.org/r/14741/#comment55380>

    create a new timestamped version of oozie sharelib


- Rohini Palaniswamy


On Nov. 8, 2013, 6:33 a.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 8, 2013, 6:33 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28680
-----------------------------------------------------------



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment55620>

    Also need to move this to AG_Install doc instead of quick start.


- Rohini Palaniswamy


On Nov. 8, 2013, 10:50 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 8, 2013, 10:50 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.

> On Nov. 11, 2013, 7:28 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 473
> > <https://reviews.apache.org/r/14741/diff/9/?file=381242#file381242line473>
> >
> >     ??

Sharelib load attempt will only once.


- Purshotam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28676
-----------------------------------------------------------


On Nov. 12, 2013, 7:56 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 12, 2013, 7:56 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28676
-----------------------------------------------------------


Just few minor comments. Javadoc needs to be updated in places based on review comments addressed. 


http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55601>

    Use caps for static variable



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55596>

    Do not add non-Javadoc comments



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55606>

    Needs updating



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55607>

    To be removed



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55605>

    Needs updating



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55604>

    To be removed



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55600>

    shareLibMap - need to rename in few other places too. Left over from old patch.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55599>

    ??



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55603>

    To be removed



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55602>

    To be removed



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml
<https://reviews.apache.org/r/14741/#comment55608>

    Keep example consistent with open source pig and hadoop. Also use hdfs:/// path prefix



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment55609>

    Example here as well


- Rohini Palaniswamy


On Nov. 8, 2013, 10:50 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 8, 2013, 10:50 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.

> On Nov. 12, 2013, 8:19 p.m., Robert Kanter wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 289
> > <https://reviews.apache.org/r/14741/diff/9/?file=381242#file381242line289>
> >
> >     Instead of using a synchronized block here, can you use the Locks service?  This way, when Oozie HA is enabled, it will automatically use the distributed locks so multiple Oozie servers won't try to update the sharelib/launcherlib at the same time.  You can use a lock named "org.apache.oozie.service.ShareLibService".  
> >     
> >     We should also put this below where the other synchronized block is.  
> >     
> >     There are probably other places where this could be a problem too, like purging the sharelib.  I think that simply surrounding it with a lock from the Locks service should take care of these cases.
> >     
> >     Also, I assume we only want one server to deal with purging the sharelib, right?  If so, you can use the #isFirstServer() method from the JobsConcurrencyService to check if the server should be doing the purge.  It always returns true for non-HA and only returns true for one server with HA.

synchronized is used for caching file from DFS. There is no update here. It used in scenario where caching of sharelib at startup failed. At the first job submission we try to cache sharelib. We synchronized the block with double check to take care of multiple job submission at same time.

We might have to think for system launcher jar. When SHIP_LAUNCHER_JAR=true server tries to copy jars from local to DFS. I believe copy of system launcher jar should be ok. With current implementation system launcher jar are always copied to DFS with new timestamp directory. Each server will be using his own version of system share lib.

We might have to think of purging, currently server delete anything older than 7 days.In case of HA, restart of 1 server might delete system share lib of long running other Oozie server.

Yup, only one server should purge the sharelib, we will use isFirstServer() for that.


- Purshotam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28752
-----------------------------------------------------------


On Nov. 12, 2013, 7:56 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 12, 2013, 7:56 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Robert Kanter <rk...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28752
-----------------------------------------------------------



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55751>

    Instead of using a synchronized block here, can you use the Locks service?  This way, when Oozie HA is enabled, it will automatically use the distributed locks so multiple Oozie servers won't try to update the sharelib/launcherlib at the same time.  You can use a lock named "org.apache.oozie.service.ShareLibService".  
    
    We should also put this below where the other synchronized block is.  
    
    There are probably other places where this could be a problem too, like purging the sharelib.  I think that simply surrounding it with a lock from the Locks service should take care of these cases.
    
    Also, I assume we only want one server to deal with purging the sharelib, right?  If so, you can use the #isFirstServer() method from the JobsConcurrencyService to check if the server should be doing the purge.  It always returns true for non-HA and only returns true for one server with HA.


- Robert Kanter


On Nov. 12, 2013, 7:56 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 12, 2013, 7:56 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28839
-----------------------------------------------------------

Ship it!


Ship It!

- Rohini Palaniswamy


On Nov. 13, 2013, 9:57 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 13, 2013, 9:57 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 1541609 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Nov. 13, 2013, 9:57 p.m.)


Review request for oozie.


Changes
-------

review comment.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).

4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
 This can be configured in oozie-site.xml as :
    <property>
        <name>oozie.service.ShareLibService.mapping.file</name>
        <value></value>
        <description>
            Sharelib mapping files contains list of key=value,
            where key is the action key and value is the DFS location of sharelib files.
        </description>
    </property>

--------------------
DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 1541609 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28836
-----------------------------------------------------------



http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java
<https://reviews.apache.org/r/14741/#comment55852>

    Please reuse the method from SharelibService instead of duplicating code


- Rohini Palaniswamy


On Nov. 13, 2013, 9:36 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 13, 2013, 9:36 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1541609 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 1541609 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Nov. 13, 2013, 9:36 p.m.)


Review request for oozie.


Changes
-------

test fix.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).

4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
 This can be configured in oozie-site.xml as :
    <property>
        <name>oozie.service.ShareLibService.mapping.file</name>
        <value></value>
        <description>
            Sharelib mapping files contains list of key=value,
            where key is the action key and value is the DFS location of sharelib files.
        </description>
    </property>

--------------------
DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1541609 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/test/java/org/apache/oozie/tools/TestOozieSharelibCLI.java 1541609 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28770
-----------------------------------------------------------

Ship it!


Ship It!

- Rohini Palaniswamy


On Nov. 12, 2013, 10:41 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 12, 2013, 10:41 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Nov. 12, 2013, 10:41 p.m.)


Review request for oozie.


Changes
-------

review comment.


We will address HA issues @https://issues.apache.org/jira/browse/OOZIE-1609.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).

4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
 This can be configured in oozie-site.xml as :
    <property>
        <name>oozie.service.ShareLibService.mapping.file</name>
        <value></value>
        <description>
            Sharelib mapping files contains list of key=value,
            where key is the action key and value is the DFS location of sharelib files.
        </description>
    </property>

--------------------
DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28761
-----------------------------------------------------------



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment55762>

    shareLibLoaded = true should be here.



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki
<https://reviews.apache.org/r/14741/#comment55766>

    ship a launcher jar to hdfs



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki
<https://reviews.apache.org/r/14741/#comment55765>

    Remove this line. It should be faster now as we copy to hdfs only once during startup



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki
<https://reviews.apache.org/r/14741/#comment55763>

    sharelib CLI.
    
    directory. While starting, server also purges sharelib directories which are


- Rohini Palaniswamy


On Nov. 12, 2013, 7:56 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 12, 2013, 7:56 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Nov. 12, 2013, 7:56 p.m.)


Review request for oozie.


Changes
-------

Review comment..


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).

4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
 This can be configured in oozie-site.xml as :
    <property>
        <name>oozie.service.ShareLibService.mapping.file</name>
        <value></value>
        <description>
            Sharelib mapping files contains list of key=value,
            where key is the action key and value is the DFS location of sharelib files.
        </description>
    </property>

--------------------
DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Nov. 8, 2013, 10:50 p.m.)


Review request for oozie.


Changes
-------

Review changes.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).

4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
 This can be configured in oozie-site.xml as :
    <property>
        <name>oozie.service.ShareLibService.mapping.file</name>
        <value></value>
        <description>
            Sharelib mapping files contains list of key=value,
            where key is the action key and value is the DFS location of sharelib files.
        </description>
    </property>

--------------------
DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Nov. 8, 2013, 6:33 a.m.)


Review request for oozie.


Changes
-------

Following changes after today's meeting
1. Tag file can be in DFS.
2. Tag file can be Hadoop supported conf file(xml, prop).
3. Should purge even if property is not set.
4. Tag file key will be prefix with Oozie.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).

4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
 This can be configured in oozie-site.xml as :
    <property>
        <name>oozie.service.ShareLibService.mapping.file</name>
        <value></value>
        <description>
            Sharelib mapping files contains list of key=value,
            where key is the action key and value is the DFS location of sharelib files.
        </description>
    </property>

--------------------
DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Nov. 7, 2013, 5:59 a.m.)


Review request for oozie.


Changes
-------

Review changes.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).

4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
 This can be configured in oozie-site.xml as :
    <property>
        <name>oozie.service.ShareLibService.mapping.file</name>
        <value></value>
        <description>
            Sharelib mapping files contains list of key=value,
            where key is the action key and value is the DFS location of sharelib files.
        </description>
    </property>

--------------------
DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.

> On Nov. 5, 2013, 6:54 p.m., Virag Kothari wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 427
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line427>
> >
> >     if path is null if all directory's are somehow malformed, then need to be handled by the caller function

This is taken care while adding path to cache.


- Purshotam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28196
-----------------------------------------------------------


On Nov. 7, 2013, 5:59 a.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 7, 2013, 5:59 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Robert Kanter <rk...@cloudera.com>.

> On Nov. 5, 2013, 6:54 p.m., Virag Kothari wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 252
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line252>
> >
> >     it might happen that this property was true during previous instance of oozie running but now its false. So need to remove the if condition. Same for the mapping file below.

FYI: when getting a boolean from a Configuration, there is a .getBoolean method you can use instead of having to do Boolean.parseBoolean


- Robert


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28196
-----------------------------------------------------------


On Nov. 1, 2013, 5:58 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 1, 2013, 5:58 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28196
-----------------------------------------------------------


minor comments


http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54860>

    instead of reading this value everytime, better read it once (constructor) and store in a variable.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54863>

    remove /// at end



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54864>

    javadoc



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54872>

    it might happen that this property was true during previous instance of oozie running but now its false. So need to remove the if condition. Same for the mapping file below.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54873>

    it might happen that this property was true during previous instance of oozie running but now its false. So need to remove the if condition. Same for the mapping file condition below.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54866>

    remove the blank line, same comment for other places in the function.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54867>

    remove



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54871>

    javadoc



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54870>

    javadoc



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54869>

    javadoc



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54868>

    if path is null if all directory's are somehow malformed, then need to be handled by the caller function


- Virag Kothari


On Nov. 1, 2013, 5:58 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 1, 2013, 5:58 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.

> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 97
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line97>
> >
> >     It does not need launcherlibPath. Needs the system lib path only. 
> >     
> >     Also log a error message if exception in purge. don't fail

exception is handled and logged at parent method.


> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 287
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line287>
> >
> >     how does this work without prefix?

This will be taken care by..

PathFilter directoryFilter = new PathFilter() {
            @Override
            public boolean accept(Path path) {
                return path.getName().startsWith(prefix);
            }
        };


> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 252
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line252>
> >
> >     Do purge irrespective of the setting.

If property is not set, then why to delete something and throw exception. 


> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 257
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line257>
> >
> >     Do purge irrespective of the setting.

If property is not set, then why to delete something and throw exception.


> On Nov. 5, 2013, 10:59 p.m., Rohini Palaniswamy wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java, line 128
> > <https://reviews.apache.org/r/14741/diff/6/?file=376492#file376492line128>
> >
> >     This code should not be removed

Discussed with Viraj, we need this method.

details @ https://issues.apache.org/jira/browse/OOZIE-1528


- Purshotam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28053
-----------------------------------------------------------


On Nov. 7, 2013, 5:59 a.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 7, 2013, 5:59 a.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Rohini Palaniswamy <ro...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review28053
-----------------------------------------------------------



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54913>

    Move creation of fs after this based on systemLibPath and move it out of for loop



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54573>

    Use a different method name like getActionShareLibJars



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54909>

    Rename to addShippedLauncherShareLibForAction



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54908>

    Revert ///



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment54948>

    Call getActionLauncherJars. return if listOfPaths is null



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54932>

    rename to launcherJarsMap and have a separate shareLibMap



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54850>

    Construct the tmp path inside the if block.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54933>

    Extract this out to a separate method shipLauncherJars



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54848>

    Make this as a local variable instead of parsing every time



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54849>

    It does not need launcherlibPath. Needs the system lib path only. 
    
    Also log a error message if exception in purge. don't fail



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54915>

    Should not be needing this on init.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54925>

    Combine into one log statement.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54917>

    launcherLibPath



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54874>

    This code should not be removed



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54919>

    Not required



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54927>

    Rename method to getActionLauncherJars



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54947>

    if shipLauncher setting is true &&



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54949>

    copyLauncherJars



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54946>

    throw the exception. do not log.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54929>

    inline this method in init() and remove this method.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54922>

    Do purge irrespective of the setting.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54923>

    Do purge irrespective of the setting.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54924>

    launcherJarLibPath is not required in method signature



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54875>

    how does this work without prefix?



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54934>

    Method can be removed



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54935>

    Call method as cacheShareLibMapping



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54936>

    Use a different map for this - shareLibMap



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54938>

    Why is type required in method signature. Can be removed



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54944>

    properties



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54943>

    PropertiesFile



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54942>

    Can remove this method



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54940>

    LauncherLibPath



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54941>

    tmpLauncherLibPath



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54937>

    getLatestLibPath



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment54876>

    have this as a local variable



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment54877>

    Remove upgrade



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment54878>

    remove "new or upgrading existing"



http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki
<https://reviews.apache.org/r/14741/#comment54879>

    Specify the format lib_<timestamp>
    
    timestamped instead of time-stamp


- Rohini Palaniswamy


On Nov. 1, 2013, 5:58 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 1, 2013, 5:58 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Nov. 1, 2013, 5:58 p.m.)


Review request for oozie.


Changes
-------

Review changes..


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).

4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
 This can be configured in oozie-site.xml as :
    <property>
        <name>oozie.service.ShareLibService.mapping.file</name>
        <value></value>
        <description>
            Sharelib mapping files contains list of key=value,
            where key is the action key and value is the DFS location of sharelib files.
        </description>
    </property>

--------------------
DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.

> On Oct. 29, 2013, 8:11 p.m., Virag Kothari wrote:
> > http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java, line 478
> > <https://reviews.apache.org/r/14741/diff/5/?file=372236#file372236line478>
> >
> >     if listOfpaths is null, then we need to have previous logic of reading the files from hdfs and adding to cache.

Sharelib is empty, trying to repopulate them.


- Purshotam


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27715
-----------------------------------------------------------


On Nov. 1, 2013, 5:58 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Nov. 1, 2013, 5:58 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/resources/oozie-default.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Virag Kothari <vi...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/#review27715
-----------------------------------------------------------



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/conf/oozie-site.xml
<https://reviews.apache.org/r/14741/#comment53834>

    uncomment and have it in oozie-default



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java
<https://reviews.apache.org/r/14741/#comment53804>

    if listOfpaths is null, then we need to have previous logic of reading the files from hdfs and adding to cache.



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment53805>

    lowercase 'o' in Oozie



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment53808>

    call if property file doesn't exist



http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java
<https://reviews.apache.org/r/14741/#comment53812>

    remove if not used



http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java
<https://reviews.apache.org/r/14741/#comment53836>

    create a new timestamp dir for upgrade


- Virag Kothari


On Oct. 28, 2013, 5:58 p.m., Purshotam Shah wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14741/
> -----------------------------------------------------------
> 
> (Updated Oct. 28, 2013, 5:58 p.m.)
> 
> 
> Review request for oozie.
> 
> 
> Bugs: OOZIE-1584
>     https://issues.apache.org/jira/browse/OOZIE-1584
> 
> 
> Repository: oozie
> 
> 
> Description
> -------
> 
> Supported features.
> -----------------
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
> server start.
> 
> 2. Admin copies sharelib to new timestamp directory and issues invalidate
> command to Oozie server. Oozie picks latest shared lib.
> 
> 3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
> sharelib jar pushed by admin(based on timestamp).
> 
> 4. Sharelib mapping files.
> Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
>  This can be configured in oozie-site.xml as :
>     <property>
>         <name>oozie.service.ShareLibService.mapping.file</name>
>         <value></value>
>         <description>
>             Sharelib mapping files contains list of key=value,
>             where key is the action key and value is the DFS location of sharelib files.
>         </description>
>     </property>
> 
> --------------------
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> 
> 
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> 
> 
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> 
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> 
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$
> 
> 
> Diffs
> -----
> 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/conf/oozie-site.xml 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
>   http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 
> 
> Diff: https://reviews.apache.org/r/14741/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Purshotam Shah
> 
>


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Oct. 28, 2013, 5:58 p.m.)


Review request for oozie.


Changes
-------

+ Sharelib mapping file support.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description (updated)
-------

Supported features.
-----------------
1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).

4. Sharelib mapping files.
Sharelib mapping file can be also configured.Configured file is a key value mapping, where key will be the action and value is DFS directory of sharelib files for action.
 This can be configured in oozie-site.xml as :
    <property>
        <name>oozie.service.ShareLibService.mapping.file</name>
        <value></value>
        <description>
            Sharelib mapping files contains list of key=value,
            where key is the action key and value is the DFS location of sharelib files.
        </description>
    </property>

--------------------
DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/conf/oozie-site.xml 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Oct. 21, 2013, 10:09 p.m.)


Review request for oozie.


Changes
-------

Review changes + Some more test-cases.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.

1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).


DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah


Re: Review Request 14741: Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

Posted by Purshotam Shah <pu...@yahoo-inc.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14741/
-----------------------------------------------------------

(Updated Oct. 21, 2013, 4:01 p.m.)


Review request for oozie.


Changes
-------

with review changes.


Bugs: OOZIE-1584
    https://issues.apache.org/jira/browse/OOZIE-1584


Repository: oozie


Description
-------

Supported features.

1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at 
server start.

2. Admin copies sharelib to new timestamp directory and issues invalidate
command to Oozie server. Oozie picks latest shared lib.

3. Oozie server doesn't auto deploy sharelib at startup. It picks the latest
sharelib jar pushed by admin(based on timestamp).


DFS after generating launcher jar by ooozie server  and sharelib by cli

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806
makebag-lm:example purushah$


Purging.
-------------
There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
directory and purging happens for both at startup.
Anything older than ( defined in property)  is purged.
We always keep 2 set of directory, purging happens if number of  sharelib
directory  > 2.


Multiple version.
-------------------
To support multiple version, sharelib will be in below format.

Sharelib will cache all versions and return list of dfs files for a particular
version.

makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
/user/purushah/share/lib/
Found 15 items
/user/purushah/share/lib/launcher_20131017092254
/user/purushah/share/lib/launcher_20131017093814
/user/purushah/share/lib/launcher_20131017094652
/user/purushah/share/lib/launcher_20131017094836
/user/purushah/share/lib/launcher_20131017095549
/user/purushah/share/lib/lib_20131017092806/pig/...
/user/purushah/share/lib/lib_20131017092806/pig_9/...
/user/purushah/share/lib/lib_20131017092806/pig_10/...
makebag-lm:example purushah$


Diffs (updated)
-----

  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/main/java/org/apache/oozie/service/ShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/core/src/test/java/org/apache/oozie/service/TestShareLibService.java 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/AG_Install.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/docs/src/site/twiki/DG_QuickStart.twiki 1532127 
  http://svn.apache.org/repos/asf/oozie/trunk/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java 1532127 

Diff: https://reviews.apache.org/r/14741/diff/


Testing
-------


Thanks,

Purshotam Shah