[jira] [Created] (OOZIE-2802) Spark action failure on Spark 2.1.0 due to duplicate sharelibs

["Satish Subhashrao Saley (JIRA)" <ji...@apache.org>]

Satish Subhashrao Saley created OOZIE-2802:
----------------------------------------------

             Summary: Spark action failure on Spark 2.1.0 due to duplicate sharelibs
                 Key: OOZIE-2802
                 URL: https://issues.apache.org/jira/browse/OOZIE-2802
             Project: Oozie
          Issue Type: Bug
            Reporter: Satish Subhashrao Saley


There are some same jars in launcher lib and sharelib. These jars are causing failures in spark action for spark 2.1.0. SPARK-18099 added an exception to be thrown if same file gets added multiple times to distributed cache.

{code}

$ hdfs dfs -ls /user/saley/share/lib/launcher_20170216150752/oozie
17/02/16 15:14:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
-rwxr-xr-x   1 saley supergroup      11871 2017-02-16 15:07 /user/saley/share/lib/launcher_20170216150752/oozie/oozie-hadoop-utils-hadoop-2-4.4.0-SNAPSHOT.jar
-rwxr-xr-x   1 saley supergroup      17514 2017-02-16 15:07 /user/saley/share/lib/launcher_20170216150752/oozie/oozie-sharelib-hcatalog-4.4.0-SNAPSHOT.jar
-rwxr-xr-x   1 saley supergroup      55092 2017-02-16 15:07 /user/saley/share/lib/launcher_20170216150752/oozie/oozie-sharelib-oozie-4.4.0-SNAPSHOT.jar



$ hdfs dfs -ls /user/saley/share/lib/lib_20170216150706/oozie
17/02/16 15:14:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 3 items
-rw-r--r--   1 saley supergroup      16046 2017-02-16 15:07 /user/saley/share/lib/lib_20170216150706/oozie/json-simple-1.1.jar
-rw-r--r--   1 saley supergroup      11871 2017-02-16 15:07 /user/saley/share/lib/lib_20170216150706/oozie/oozie-hadoop-utils-hadoop-2-4.4.0-SNAPSHOT.jar
-rw-r--r--   1 saley supergroup      55092 2017-02-16 15:07 /user/saley/share/lib/lib_20170216150706/oozie/oozie-sharelib-oozie-4.4.0-SNAPSHOT.jar

{code}


The file with same name can't be in twice in any of *--files , --archives, --jars* (ie one file in both --files and --jars) and can't be in --files twice and can't be in --archives twice. It is allowed to be in --jars twice.

Possible solutions:
1. While specifying the --files  options, filter out the duplicate jars
2. Separate out jars and files. Use --jars options to distribute jars and --files for files. Also remember to remove duplicates from --files, and make sure that there is nothing common in --jars and --files.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)