You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Bohdan Demydov (JIRA)" <ji...@apache.org> on 2018/01/02 13:48:00 UTC
[jira] [Updated] (LIVY-430) Livy always run on local mode
[ https://issues.apache.org/jira/browse/LIVY-430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bohdan Demydov updated LIVY-430:
--------------------------------
Description:
I am trying to run Pyspark (or Spark) job via Livy server with "spark.master=yarn".
*What I have done:*
1) In *spark-defaults.conf:*
spark.master yarn
spark.submit.deployMode client
2) In *livy.conf*:
livy.spark.master = yarn
livy.spark.deployMode = client
3) I send request via CURL *with "conf": {"spark.master": "yarn"}
*
Example:
curl -X POST -H "Content-Type: application/json" localhost:8998/batches --data '{"file": "hdfs:///user/grzegorz/hello-world.py", "name": "MY", "conf": {"spark.master": "yarn"} }'
{"id":3,"state":"running","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: "]}
*And what I am always getting in logs:*
18/01/02 14:45:07.880 qtp1758624236-28 INFO BatchSession$: Creating batch session 3: [owner: null, request: [proxyUser: None, file: hdfs:///user/grzegorz/hello-world.py, name: MY, conf: spark.master -> yarn]]
18/01/02 14:45:07.883 qtp1758624236-28 INFO SparkProcessBuilder: Running '/usr/local/share/spark/spark-2.0.2/bin/spark-submit' '--name' 'MY' '--conf'* 'spark.master=local' *'hdfs:///user/grzegorz/hello-world.py'
was:
I am trying to run Pyspark (or Spark) job via Livy server with "spark.master=local".
*What I have done:*
1) In *spark-defaults.conf:*
spark.master yarn
spark.submit.deployMode client
2) In *livy.conf*:
livy.spark.master = yarn
livy.spark.deployMode = client
3) I send request via CURL *with "conf": {"spark.master": "yarn"}
*
Example:
curl -X POST -H "Content-Type: application/json" localhost:8998/batches --data '{"file": "hdfs:///user/grzegorz/hello-world.py", "name": "MY", "conf": {"spark.master": "yarn"} }'
{"id":3,"state":"running","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: "]}
*And what I am always getting in logs:*
18/01/02 14:45:07.880 qtp1758624236-28 INFO BatchSession$: Creating batch session 3: [owner: null, request: [proxyUser: None, file: hdfs:///user/grzegorz/hello-world.py, name: MY, conf: spark.master -> yarn]]
18/01/02 14:45:07.883 qtp1758624236-28 INFO SparkProcessBuilder: Running '/usr/local/share/spark/spark-2.0.2/bin/spark-submit' '--name' 'MY' '--conf'* 'spark.master=local' *'hdfs:///user/grzegorz/hello-world.py'
> Livy always run on local mode
> -----------------------------
>
> Key: LIVY-430
> URL: https://issues.apache.org/jira/browse/LIVY-430
> Project: Livy
> Issue Type: Bug
> Affects Versions: 0.3
> Environment: Spark 2.2.0
> Hadoop 2.7.4
> Reporter: Bohdan Demydov
>
> I am trying to run Pyspark (or Spark) job via Livy server with "spark.master=yarn".
> *What I have done:*
> 1) In *spark-defaults.conf:*
> spark.master yarn
> spark.submit.deployMode client
> 2) In *livy.conf*:
> livy.spark.master = yarn
> livy.spark.deployMode = client
> 3) I send request via CURL *with "conf": {"spark.master": "yarn"}
> *
> Example:
> curl -X POST -H "Content-Type: application/json" localhost:8998/batches --data '{"file": "hdfs:///user/grzegorz/hello-world.py", "name": "MY", "conf": {"spark.master": "yarn"} }'
> {"id":3,"state":"running","appId":null,"appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: "]}
> *And what I am always getting in logs:*
> 18/01/02 14:45:07.880 qtp1758624236-28 INFO BatchSession$: Creating batch session 3: [owner: null, request: [proxyUser: None, file: hdfs:///user/grzegorz/hello-world.py, name: MY, conf: spark.master -> yarn]]
> 18/01/02 14:45:07.883 qtp1758624236-28 INFO SparkProcessBuilder: Running '/usr/local/share/spark/spark-2.0.2/bin/spark-submit' '--name' 'MY' '--conf'* 'spark.master=local' *'hdfs:///user/grzegorz/hello-world.py'
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)