[jira] [Comment Edited] (OOZIE-2767) Oozie 4.3.0 with spark 2.0 run spark R job on yarn cluster mode failed

["Robert Kanter (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797794#comment-15797794 ] 

Robert Kanter edited comment on OOZIE-2767 at 1/4/17 8:27 PM:
--------------------------------------------------------------

Hi [~Dongying Jiao],

Spark 2 has reworked a remarkable subset of the properties used by [{{SparkSubmit.scala}}|http://github.mtv.cloudera.com/CDH/spark/blob/spark2-2.0.0/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala], so apart from {{master}} parameter there are plenty of changes. Please look at the attached {{bash}} files using Spark 1's and Spark 2's {{spark-submit}} scripts.

Oozie 4.3.0 does not support Spark 2. It only supports Spark 1.6.x. In Oozie 5.0.0, we're planning to support Spark 2. 

Regards,

Andras


was (Author: andras.piros):
Hi [~Dongying Jiao],

Spark 2 has reworked a remarkable subset of the properties used by [{{SparkSubmit.scala}}|http://github.mtv.cloudera.com/CDH/spark/blob/spark2-2.0.0/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala], so apart from {{master}} parameter there are plenty of changes. Please look at the attached {{bash}} files using Spark 1's and Spark 2's {{spark-submit}} scripts.

Oozie 4.3.0 does not support Spark 2. It only supports Spark 1.6.x. Oozie 5.0.0 (packaged within CDH 6) will support Spark 2 (and Spark 2 only). There are plans for Oozie 4.4.x to support Spark 2 within CDH 5.x, but cannot confirm any exact release dates / version numbers.

Regards,

Andras

> Oozie 4.3.0 with spark 2.0 run spark R job on yarn cluster mode failed 
> -----------------------------------------------------------------------
>
>                 Key: OOZIE-2767
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2767
>             Project: Oozie
>          Issue Type: Bug
>          Components: action
>    Affects Versions: 4.3.0
>            Reporter: Dongying Jiao
>         Attachments: spark1.sh, spark2.sh
>
>
> Erro seen from yarn log:
> Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
> Error: sparkr.zip does not exist for R application in YARN mode.
> Run with --help for usage help or --verbose for debug output
> Intercepting System.exit(1)
> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [1]
> End of LogType:stderr



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)