[jira] [Updated] (OOZIE-3042) Allow isolation of workflow action dependencies

["Robert Kanter (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OOZIE-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Kanter updated OOZIE-3042:
---------------------------------
    Issue Type: New Feature  (was: Bug)

> Allow isolation of workflow action dependencies 
> ------------------------------------------------
>
>                 Key: OOZIE-3042
>                 URL: https://issues.apache.org/jira/browse/OOZIE-3042
>             Project: Oozie
>          Issue Type: New Feature
>    Affects Versions: 4.3.0, 5.0.0
>            Reporter: Attila Sasvari
>
> Currently JAR files stored in the lib/ subdirectory of the application directory are automatically added in the classpath of Hadoop jobs started by Oozie. There can be problems if a workflow includes multiple actions and classes are not compatible.
> For example, I recently experienced an issue that occurred when my workflow included a Spark action after a Java action. A jar file with classes my Java action depended on contained classes that caused runtime issues for the Spark action. I enabled verbose class loading for the Spark driver and also for the Oozie launcher and its application master to see what was going on. Incompatible classes were loaded from the classpath when Oozie tried to run the second action. (Both actions were able to run without error in separate actions. )
> The problem is that Oozie puts everything to the classpath in the workflow {{lib}} directory, even if it might cause problem for other action. 
> The workflow {{lib}} directory could contain subdirs with the name of the workflow actions. These wf actions could store their dependencies in those directories. Say if my workflow has A1 and A2 actions there could be lib/A1 and lib/A2.
> {{JavaActionExecutor}} could look for the presence of those directories based on the action name.
> If there is such a directory it would build the classpath accordingly -> first jars would come from {{lib/A1/}} if it is the A1 action to be executed. jars in the {{lib}} directory could go to the end of the classpath. Those can be considered common for all the workflow actions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)