You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by ah...@apache.org on 2016/11/16 14:30:03 UTC
zeppelin git commit: [Docs] Add note on configuration of spark-yarn
mode on docker
Repository: zeppelin
Updated Branches:
refs/heads/master 520228fec -> e0930570d
[Docs] Add note on configuration of spark-yarn mode on docker
### What is this PR for?
This adds note to configuration to make clear about `sparkmaster` hostname defined in `/etc/hosts`
### What type of PR is it?
Improvement | Documentation
### Todos
* [x] - add note
### What is the Jira issue?
n/a
### How should this be tested?
see documentation
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Khalid Huseynov <kh...@gmail.com>
Closes #1636 from khalidhuseynov/docs/spark-yarn-add-note-hostname and squashes the following commits:
9648e08 [Khalid Huseynov] address feedback, fix path
78e7907 [Khalid Huseynov] add note about sparkmaster to each section
887b499 [Khalid Huseynov] update docker file path yarn mode
9e4ad68 [Khalid Huseynov] add sparkmaster hostname note
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/e0930570
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/e0930570
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/e0930570
Branch: refs/heads/master
Commit: e0930570df6bfeefb4f283f6624ce2ce6caf0fe0
Parents: 520228f
Author: Khalid Huseynov <kh...@gmail.com>
Authored: Wed Nov 16 15:44:32 2016 +0900
Committer: ahyoungryu <ah...@apache.org>
Committed: Wed Nov 16 15:29:30 2016 +0100
----------------------------------------------------------------------
docs/install/spark_cluster_mode.md | 16 +++++++++++-----
1 file changed, 11 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/e0930570/docs/install/spark_cluster_mode.md
----------------------------------------------------------------------
diff --git a/docs/install/spark_cluster_mode.md b/docs/install/spark_cluster_mode.md
index d4c864a..b848c96 100644
--- a/docs/install/spark_cluster_mode.md
+++ b/docs/install/spark_cluster_mode.md
@@ -23,17 +23,18 @@ limitations under the License.
<div id="toc"></div>
-## Overview
+## Overview
[Apache Spark](http://spark.apache.org/) has supported three cluster manager types([Standalone](http://spark.apache.org/docs/latest/spark-standalone.html), [Apache Mesos](http://spark.apache.org/docs/latest/running-on-mesos.html) and [Hadoop YARN](http://spark.apache.org/docs/latest/running-on-yarn.html)) so far.
This document will guide you how you can build and configure the environment on 3 types of Spark cluster manager with Apache Zeppelin using [Docker](https://www.docker.com/) scripts.
So [install docker](https://docs.docker.com/engine/installation/) on the machine first.
## Spark standalone mode
[Spark standalone](http://spark.apache.org/docs/latest/spark-standalone.html) is a simple cluster manager included with Spark that makes it easy to set up a cluster.
-You can simply set up Spark standalone environment with below steps.
+You can simply set up Spark standalone environment with below steps.
> **Note :** Since Apache Zeppelin and Spark use same `8080` port for their web UI, you might need to change `zeppelin.server.port` in `conf/zeppelin-site.xml`.
+
### 1. Build Docker file
You can find docker script files under `scripts/docker/spark-cluster-managers`.
@@ -52,9 +53,11 @@ docker run -it \
-p 8081:8081 \
-h sparkmaster \
--name spark_standalone \
-spark_standalone bash;
+spark_standalone bash;
```
+Note that `sparkmaster` hostname used here to run docker container should be defined in your `/etc/hosts`.
+
### 3. Configure Spark interpreter in Zeppelin
Set Spark master as `spark://<hostname>:7077` in Zeppelin **Interpreters** setting page.
@@ -81,7 +84,7 @@ You can simply set up [Spark on YARN](http://spark.apache.org/docs/latest/runnin
You can find docker script files under `scripts/docker/spark-cluster-managers`.
```
-cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_yarn
+cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_yarn_cluster
docker build -t "spark_yarn" .
```
@@ -111,6 +114,8 @@ docker run -it \
spark_yarn bash;
```
+Note that `sparkmaster` hostname used here to run docker container should be defined in your `/etc/hosts`.
+
### 3. Verify running Spark on YARN.
You can simply verify the processes of Spark and YARN are running well in Docker with below command.
@@ -172,6 +177,8 @@ docker run --net=host -it \
spark_mesos bash;
```
+Note that `sparkmaster` hostname used here to run docker container should be defined in your `/etc/hosts`.
+
### 3. Verify running Spark on Mesos.
You can simply verify the processes of Spark and Mesos are running well in Docker with below command.
@@ -201,4 +208,3 @@ Don't forget to set Spark `master` as `mesos://127.0.1.1:5050` in Zeppelin **Int
After running a single paragraph with Spark interpreter in Zeppelin, browse `http://<hostname>:5050/#/frameworks` and check Zeppelin application is running well or not.
<img src="../assets/themes/zeppelin/img/docs-img/mesos_frameworks.png" />
-