[jira] [Updated] (OOZIE-1619) Enhancing sharelib support

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OOZIE-1619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated OOZIE-1619:
--------------------------------------

    Summary: Enhancing sharelib support  (was: Sharelib support.)

Currently Oozie provides support for user sharing action (pig/hive/hbase/hcatalog/distcp) jars through sharelib. 

Current limitations:
   - The sharelib directory is created under workflow system lib path. During oozie upgrade or a pig upgrade, the sharelib directory is deleted and recreated. This makes all the existing running jobs fail because Hadoop fails any jobs if the jars shipped using Distributed Cache have changed before the job completes. Having to rerun those jobs wastes valuable cluster resources and causes sla misses.
   
Features:

Sharelib Creation:
  -  Admin runs sharelib create command as usual. But the command copies the sharelib contents into a new timestamped version of the sharelib directory (lib_<timestamp>) under the system lib path directory. Previously the contents where copied directly under system lib path directory.
  -  Oozie on startup picks up the latest sharelib directory. It purges sharelib directories older than 7 days (configurable) except the previous latest one. 
  -  Contents of sharelib can viewed by running "oozie admin -shareliblist" command.
  
Sharelib Updation:
  - While oozie server is up, if there are newer versions of pig or hive jars available it should be possible for oozie to switch to the latest one without restarting oozie. This is done by admin running the sharelib create command again with the newer version of sharelib. Then "oozie admin -sharelibupdate" command can be issued to the server to make it pick up the latest sharelib. If it is a HA environment it will be propagated to all servers. As the previous sharelib is still not deleted, jobs which started with those jars will run successfully to completion.
  
Sharelib Meta File:
  - This is an alternative to the sharelib directory. If you already have pig,hive,etc jars installed in some hdfs locations, then this property file can be used to specify those locations. Newer versions can be picked up by updating this file and invoking "oozie admin -sharelibupdate". 
  
Shipping Launcher jars:
  - By default, it is expected that the sharelib for particular action contains the oozie launcher jar as well. For eg: pig action sharelib should have oozie-sharelib-pig-<oozie version>.jar. This is automatically there when you use the sharelib tar file created by oozie. If you have your own setup with just the pig jars, then you can set oozie.action.ship.launcher.jar=true. In that case, oozie will automatically ship the launcher jars (from the tomcat webapp WEB-INF/lib directory) to hdfs on startup into launcher_<timestamp> directory under the system lib path directory. When launching jobs, it includes them in distributed cache for corresponding action. Purging of older launcher_<timestamp> directories is done during startup similar to lib_<timestamp> directories.

> Enhancing sharelib support
> --------------------------
>
>                 Key: OOZIE-1619
>                 URL: https://issues.apache.org/jira/browse/OOZIE-1619
>             Project: Oozie
>          Issue Type: Improvement
>            Reporter: purshotam shah
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)