You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Yeshwanth Jagini <ye...@yotabitesllc.com> on 2017/05/04 19:13:27 UTC
Unable to run Zeppelin Spark on YARN
Hi we are running cloudera CDH 5.9.1 .
while setting up zeppelin, i followed the documentation on website and
specified following options
export ZEPPELIN_JAVA_OPTS="-Dhadoop.version=2.6.0-cdh5.9.1"
# Additional jvm options. for example, export
ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"
export
SPARK_HOME="/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/spark"
# (required) When it is defined, load it instead of
Zeppelin embedded Spark libraries
export SPARK_SUBMIT_OPTIONS="--master yarn --deploy-mode client"
# (optional) extra options to pass to spark submit. eg)
"--driver-memory 512M --executor-memory 1G".
export SPARK_APP_NAME=Zeppelin # (optional) The
name of spark application.
export HADOOP_CONF_DIR=/etc/hadoop/conf #
yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.
export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER}
bash -c' # Optional, when user want to run interpreter as end web
user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '
when running spark notebook, spark-submit is running in local mode and i
cannot see the application in yarn resource manager.
is there any other configuration i am missing?
Thanks,
Yeshwanth Jagini
Re: Unable to run Zeppelin Spark on YARN
Posted by Jongyoul Lee <jo...@gmail.com>.
Hi,
"--master yarn --deploy-mode client" will be overridden when create spark
context by SparkInterpreter In zeppelin. You have to set those values in
interpreter setting page
Regards,
Jongyoul
On Fri, May 5, 2017 at 8:33 AM, Jianfeng (Jeff) Zhang <
jzhang@hortonworks.com> wrote:
>
> Could you try set yarn-client in interpreter setting page ?
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Yeshwanth Jagini <ye...@yotabitesllc.com>
> Reply-To: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
> Date: Friday, May 5, 2017 at 3:13 AM
> To: "users@zeppelin.apache.org" <us...@zeppelin.apache.org>
> Subject: Unable to run Zeppelin Spark on YARN
>
> Hi we are running cloudera CDH 5.9.1 .
>
> while setting up zeppelin, i followed the documentation on website and
> specified following options
>
> export ZEPPELIN_JAVA_OPTS="-Dhadoop.version=2.6.0-cdh5.9.1"
> # Additional jvm options. for example, export
> ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"
>
> export SPARK_HOME="/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/spark"
> # (required) When it is defined, load it instead
> of Zeppelin embedded Spark libraries
> export SPARK_SUBMIT_OPTIONS="--master yarn --deploy-mode client"
> # (optional) extra options to pass to spark submit. eg)
> "--driver-memory 512M --executor-memory 1G".
> export SPARK_APP_NAME=Zeppelin # (optional) The
> name of spark application.
>
> export HADOOP_CONF_DIR=/etc/hadoop/conf #
> yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.
>
> export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER}
> bash -c' # Optional, when user want to run interpreter as end web
> user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '
>
> when running spark notebook, spark-submit is running in local mode and i
> cannot see the application in yarn resource manager.
> is there any other configuration i am missing?
>
>
> Thanks,
> Yeshwanth Jagini
>
--
이종열, Jongyoul Lee, 李宗烈
http://madeng.net
Re: Unable to run Zeppelin Spark on YARN
Posted by "Jianfeng (Jeff) Zhang" <jz...@hortonworks.com>.
Could you try set yarn-client in interpreter setting page ?
Best Regard,
Jeff Zhang
From: Yeshwanth Jagini <ye...@yotabitesllc.com>>
Reply-To: "users@zeppelin.apache.org<ma...@zeppelin.apache.org>" <us...@zeppelin.apache.org>>
Date: Friday, May 5, 2017 at 3:13 AM
To: "users@zeppelin.apache.org<ma...@zeppelin.apache.org>" <us...@zeppelin.apache.org>>
Subject: Unable to run Zeppelin Spark on YARN
Hi we are running cloudera CDH 5.9.1 .
while setting up zeppelin, i followed the documentation on website and specified following options
export ZEPPELIN_JAVA_OPTS="-Dhadoop.version=2.6.0-cdh5.9.1" # Additional jvm options. for example, export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"
export SPARK_HOME="/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/spark" # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries
export SPARK_SUBMIT_OPTIONS="--master yarn --deploy-mode client" # (optional) extra options to pass to spark submit. eg) "--driver-memory 512M --executor-memory 1G".
export SPARK_APP_NAME=Zeppelin # (optional) The name of spark application.
export HADOOP_CONF_DIR=/etc/hadoop/conf # yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.
export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c' # Optional, when user want to run interpreter as end web user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '
when running spark notebook, spark-submit is running in local mode and i cannot see the application in yarn resource manager.
is there any other configuration i am missing?
Thanks,
Yeshwanth Jagini