[jira] [Updated] (OOZIE-2787) Oozie distributes application jar twice making the spark job fail

["Satish Subhashrao Saley (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/OOZIE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Satish Subhashrao Saley updated OOZIE-2787:
-------------------------------------------

Cherry-picked from master to branch-4.3.


> Oozie distributes application jar twice making the spark job fail
> -----------------------------------------------------------------
>
>                 Key: OOZIE-2787
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2787
>             Project: Oozie
>          Issue Type: Bug
>            Reporter: Satish Subhashrao Saley
>            Assignee: Satish Subhashrao Saley
>             Fix For: 4.3.1
>
>         Attachments: OOZIE-2787-1.patch, OOZIE-2787-2.patch, OOZIE-2787-3.patch, OOZIE-2787-4.patch, OOZIE-2787-5.patch, OOZIE-2787-amend-1.patch, OOZIE-2787-amend-2.patch, OOZIE-2787-amend-3.patch, OOZIE-2787-amend-4.patch, OOZIE-2787-amend-5.patch
>
>
> Oozie adds the application jar to the list of files to be uploaded to distributed cache. Since this gets added twice, the job fails. This is observed from spark 2.1.0 which introduces a check for same file and fails the job.
> {code}
> --master
> yarn
> --deploy-mode
> cluster
> --name
> oozieSparkStarter
> --class
> ScalaWordCount
> --queue 
> default
> --conf
> spark.executor.extraClassPath=$PWD/*
> --conf
> spark.driver.extraClassPath=$PWD/*
> --conf
> spark.executor.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.driver.extraJavaOptions=-Dlog4j.configuration=spark-log4j.properties
> --conf
> spark.yarn.security.tokens.hive.enabled=false
> --conf
> spark.yarn.security.tokens.hbase.enabled=false
> --files
> hdfs://mycluster.com/user/saley/oozie/apps/sparkapp/lib/spark-example.jar
> --properties-file
> spark-defaults.conf
> --verbose
> spark-example.jar
> samplefile.txt
> output
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)