You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by ah...@apache.org on 2016/11/16 14:30:03 UTC

zeppelin git commit: [Docs] Add note on configuration of spark-yarn mode on docker

Repository: zeppelin
Updated Branches:
  refs/heads/master 520228fec -> e0930570d


[Docs] Add note on configuration of spark-yarn mode on docker

### What is this PR for?
This adds note to configuration to make clear about `sparkmaster` hostname defined in `/etc/hosts`

### What type of PR is it?
Improvement | Documentation

### Todos
* [x] - add note

### What is the Jira issue?
n/a

### How should this be tested?
see documentation

### Screenshots (if appropriate)

### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no

Author: Khalid Huseynov <kh...@gmail.com>

Closes #1636 from khalidhuseynov/docs/spark-yarn-add-note-hostname and squashes the following commits:

9648e08 [Khalid Huseynov] address feedback, fix path
78e7907 [Khalid Huseynov] add note about sparkmaster to each section
887b499 [Khalid Huseynov] update docker file path yarn mode
9e4ad68 [Khalid Huseynov] add sparkmaster hostname note


Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/e0930570
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/e0930570
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/e0930570

Branch: refs/heads/master
Commit: e0930570df6bfeefb4f283f6624ce2ce6caf0fe0
Parents: 520228f
Author: Khalid Huseynov <kh...@gmail.com>
Authored: Wed Nov 16 15:44:32 2016 +0900
Committer: ahyoungryu <ah...@apache.org>
Committed: Wed Nov 16 15:29:30 2016 +0100

----------------------------------------------------------------------
 docs/install/spark_cluster_mode.md | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/zeppelin/blob/e0930570/docs/install/spark_cluster_mode.md
----------------------------------------------------------------------
diff --git a/docs/install/spark_cluster_mode.md b/docs/install/spark_cluster_mode.md
index d4c864a..b848c96 100644
--- a/docs/install/spark_cluster_mode.md
+++ b/docs/install/spark_cluster_mode.md
@@ -23,17 +23,18 @@ limitations under the License.
 
 <div id="toc"></div>
 
-## Overview 
+## Overview
 [Apache Spark](http://spark.apache.org/) has supported three cluster manager types([Standalone](http://spark.apache.org/docs/latest/spark-standalone.html), [Apache Mesos](http://spark.apache.org/docs/latest/running-on-mesos.html) and [Hadoop YARN](http://spark.apache.org/docs/latest/running-on-yarn.html)) so far.
 This document will guide you how you can build and configure the environment on 3 types of Spark cluster manager with Apache Zeppelin using [Docker](https://www.docker.com/) scripts.
 So [install docker](https://docs.docker.com/engine/installation/) on the machine first.
 
 ## Spark standalone mode
 [Spark standalone](http://spark.apache.org/docs/latest/spark-standalone.html) is a simple cluster manager included with Spark that makes it easy to set up a cluster.
-You can simply set up Spark standalone environment with below steps. 
+You can simply set up Spark standalone environment with below steps.
 
 > **Note :** Since Apache Zeppelin and Spark use same `8080` port for their web UI, you might need to change `zeppelin.server.port` in `conf/zeppelin-site.xml`.
 
+
 ### 1. Build Docker file
 You can find docker script files under `scripts/docker/spark-cluster-managers`.
 
@@ -52,9 +53,11 @@ docker run -it \
 -p 8081:8081 \
 -h sparkmaster \
 --name spark_standalone \
-spark_standalone bash; 
+spark_standalone bash;
 ```
 
+Note that `sparkmaster` hostname used here to run docker container should be defined in your `/etc/hosts`.
+
 ### 3. Configure Spark interpreter in Zeppelin
 Set Spark master as `spark://<hostname>:7077` in Zeppelin **Interpreters** setting page.
 
@@ -81,7 +84,7 @@ You can simply set up [Spark on YARN](http://spark.apache.org/docs/latest/runnin
 You can find docker script files under `scripts/docker/spark-cluster-managers`.
 
 ```
-cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_yarn
+cd $ZEPPELIN_HOME/scripts/docker/spark-cluster-managers/spark_yarn_cluster
 docker build -t "spark_yarn" .
 ```
 
@@ -111,6 +114,8 @@ docker run -it \
  spark_yarn bash;
 ```
 
+Note that `sparkmaster` hostname used here to run docker container should be defined in your `/etc/hosts`.
+
 ### 3. Verify running Spark on YARN.
 
 You can simply verify the processes of Spark and YARN are running well in Docker with below command.
@@ -172,6 +177,8 @@ docker run --net=host -it \
 spark_mesos bash;
 ```
 
+Note that `sparkmaster` hostname used here to run docker container should be defined in your `/etc/hosts`.
+
 ### 3. Verify running Spark on Mesos.
 
 You can simply verify the processes of Spark and Mesos are running well in Docker with below command.
@@ -201,4 +208,3 @@ Don't forget to set Spark `master` as `mesos://127.0.1.1:5050` in Zeppelin **Int
 After running a single paragraph with Spark interpreter in Zeppelin, browse `http://<hostname>:5050/#/frameworks` and check Zeppelin application is running well or not.
 
 <img src="../assets/themes/zeppelin/img/docs-img/mesos_frameworks.png" />
-