[jira] [Commented] (OOZIE-1584) Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.

["Josh Spiegel (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919648#comment-13919648 ] 

Josh Spiegel commented on OOZIE-1584:
-------------------------------------

This enhancement changes the behavior of the sharelib in two ways not mentioned:
(1) Non-jar files in the sharelib are no longer referenced by symlinks in the working directory of the task
(2) The sharelib path is now searched recursively

For example, if I add native libraries in the sharelib, they will no longer show up in the working directory of the task after this change.  

Also related to (1), for this part of the change:
https://reviews.apache.org/r/14741/diff/#0.22

    > DistributedCache.addFileToClassPath(actionLibPath, conf, fs);
    > DistributedCache.createSymlink(conf);

Is it meaningful to call createSymlink(conf) in this case?  My understanding is that files only get symlinked when there is a uri fragment and there isn't in this case.

> Setup sharelib using script and pickup latest(honor ship.launcher) and remove DFS dependency at startup.
> --------------------------------------------------------------------------------------------------------
>
>                 Key: OOZIE-1584
>                 URL: https://issues.apache.org/jira/browse/OOZIE-1584
>             Project: Oozie
>          Issue Type: Sub-task
>            Reporter: purshotam shah
>            Assignee: purshotam shah
>             Fix For: trunk
>
>         Attachments: OOZIE-1584_v2.patch, OOZIE-1584_v3.patch, OOZIE-1584_v4.patch, OOZIE-1584_v5.patch, OOZIE-1584_v6.patch
>
>
> Supported features.
> 1. If Oozie.action.ship.launcher.jar = true then, Oozie auto-deploy jar at server starts.
> 2. Admin copies sharelib to new timestamp directory(using sharelibCLI) and issues invalidate command(using oozieCLI) to Oozie server. Oozie picks latest shared lib.
> 3.Oozie server doesn't auto deploy sharelib at startup. It picks the latest sharelib jar pushed by admin(based on timestamp).
> DFS after generating launcher jar by ooozie server  and sharelib by cli
> -------------------
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806
> makebag-lm:example purushah$
> Purging.
> -------------
> There are two set( launcher_ for launcher jars and Lib_ for shared lib) of
> directory and purging happens for both at startup.
> Anything older than ( defined in property)  is purged.
> We always keep 2 set of directory, purging happens if number of  sharelib
> directory  > 2.
> Multiple version.
> -------------------
> To support multiple version, sharelib will be in below format.
> Sharelib will cache all versions and return list of dfs files for a particular
> version.
> makebag-lm:example purushah$ /var/hadoop-1.2.1/bin/hadoop fs -ls
> /user/purushah/share/lib/
> Found 15 items
> /user/purushah/share/lib/launcher_20131017092254
> /user/purushah/share/lib/launcher_20131017093814
> /user/purushah/share/lib/launcher_20131017094652
> /user/purushah/share/lib/launcher_20131017094836
> /user/purushah/share/lib/launcher_20131017095549
> /user/purushah/share/lib/lib_20131017092806/pig/...
> /user/purushah/share/lib/lib_20131017092806/pig_9/...
> /user/purushah/share/lib/lib_20131017092806/pig_10/...
> makebag-lm:example purushah$



--
This message was sent by Atlassian JIRA
(v6.2#6252)