Can I control the files in my oozie action's distcache?

["Aaron.Dossett" <Aa...@target.com>]

Hi,

I'm running an oozie job that executes a java action using HDP 2.3.4.  When this action runs, it has a lot of files in its "current directory" (see below for a full list).  Some of these files are there because I'm shipping them with the action (e.g., myapp.properties).  Many of those jar files are present whether or not I have "oozie.use.system.libpath" set to true or false in the java action.  Because they're in my current directory, they are very high up in my action's class path and causing some problems.  For example, jackson-*-2.2.3.jar or guava-11.0.2.jar.  I understand that they may need to be SOMEWHERE in the class path, because they are core hadoop dependencies, but they are present in other places too:


/grid/5/hadoop/yarn/local/usercache/SVBIG01S/appcache/application_1458846497308_8586/container_e3048_1458846497308_8586_01_000002/mr-framework/hadoop/share/hadoop/common/lib/jackson-core-2.2.3.jar


/grid/5/hadoop/yarn/local/usercache/SVBIG01S/appcache/application_1458846497308_8586/container_e3048_1458846497308_8586_01_000002/mr-framework/hadoop/share/hadoop/common/lib/guava-11.0.2.jar

For example, I want to use a different version of jackson and have it included in my utilities-1.2.0-SNAPSHOT.jar, but it shares the class path precedence with the jackson-*.jar files that are in the current directory and sometimes those get loaded.

Can I:

  *   keep these extra jar files out of my current directory entirely (my preferred solution)
  *   failing that, make sure that the jars I provide have the absolute highest class path priority (acceptable solution)

Thanks! -Aaron



Files in current dir:/grid/5/hadoop/yarn/local/usercache/MYID/appcache/application_1458846497308_8586/container_e3048_1458846497308_8586_01_000002/.
======================
File: hadoop-azure-2.7.1.2.3.4.0-3485.jar
File: jackson-databind-2.2.3.jar
File: json-simple-1.1.jar
Dir: tmp
File: hdfs-site.xml
File: job.xml
File: storm-hive-ddl-1.2.0-SNAPSHOT.jar
File: aws-java-sdk-1.7.4.jar
File: oozie-sharelib-oozie-4.2.0.2.3.4.0-3485.jar
File: action.xml
File: joda-time-2.1.jar
File: jaas.conf
File: hive-jdbc.jar
File: foundation-builder-1.2.0-SNAPSHOT.jar
File: guava-11.0.2.jar
File: container_tokens
Dir: mr-framework
  Dir: hadoop
File: azure-storage-2.2.0.jar
File: launch_container.sh
File: jackson-annotations-2.2.3.jar
File: firefly.properties
File: .action.xml.crc
File: myapp.properties
File: propagation-conf.xml
File: .job.xml.crc
File: core-site.xml
File: commons-lang3-3.3.2.jar
File: utilities-1.2.0-SNAPSHOT.jar
File: hadoop-aws-2.7.1.2.3.4.0-3485.jar
File: jackson-core-2.2.3.jar
File: oozie-hadoop-utils-hadoop-2-4.2.0.2.3.4.0-3485.jar