[jira] [Commented] (OOZIE-2767) Oozie 4.3.0 with spark 2.0 run spark R job on yarn cluster mode failed

["Dongying Jiao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/OOZIE-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15797279#comment-15797279 ] 

Dongying Jiao commented on OOZIE-2767:
--------------------------------------

Workflow is:
<workflow-app xmlns='uri:oozie:workflow:0.5' name='testSparkR'>
<global>
            <configuration>
                <property>
                    <name>oozie.launcher.yarn.app.mapreduce.am.env</name>
                    <value>SPARK_HOME=/usr/service/spark2</value>
                </property>
            </configuration>
</global>
<start to="sparkAction"/>
    <action name="sparkAction">
        <spark xmlns="uri:oozie:spark-action:0.1">
                <job-tracker>${jobTracker}</job-tracker>
                <name-node>${nameNode}</name-node>
                <master>${master}</master>
                <name>testSparkR</name>
                <jar>${nameNode}/user/oozie/sparkR/dataframe.R</jar>
         <spark-opts>--conf spark.driver.extraJavaOptions=-Dcluster.version=4.3.0 --conf spark.yarn.archive=hdfs://server1:8020/platform/apps/spark2/spark2-iop-yarn-archive.tar.gz</spark-opts>
         </spark>
      <ok to="end"/>
      <error to="fail"/> 
  </action>
  <kill name="fail">
        <message>Workflow failed, error
        message[${wf:errorMessage(wf:lastErrorNode())}]
        </message>
  </kill>
  <end name="end"/> 
</workflow-app>

job.properties:
nameNode=hdfs://server1:8020
jobTracker=server2:8050
master=yarn-cluster
queueName=default
oozie.use.system.libpath=true
oozie.wf.application.path=/user/oozie/sparkR

The R script is from spark2 examples "dataframe.R".
sparkr.zip is put in workflow lib dir.

> Oozie 4.3.0 with spark 2.0 run spark R job on yarn cluster mode failed 
> -----------------------------------------------------------------------
>
>                 Key: OOZIE-2767
>                 URL: https://issues.apache.org/jira/browse/OOZIE-2767
>             Project: Oozie
>          Issue Type: Bug
>          Components: action
>    Affects Versions: 4.3.0
>            Reporter: Dongying Jiao
>
> Erro seen from yarn log:
> Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
> Error: sparkr.zip does not exist for R application in YARN mode.
> Run with --help for usage help or --verbose for debug output
> Intercepting System.exit(1)
> Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [1]
> End of LogType:stderr



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)