You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by fe...@apache.org on 2017/10/24 06:25:12 UTC
zeppelin git commit: [ZEPPELIN-2949] Allow custom Spark UI URL
Repository: zeppelin
Updated Branches:
refs/heads/master 84cb4b5fb -> c45128e79
[ZEPPELIN-2949] Allow custom Spark UI URL
### What is this PR for?
Allow an override of default Spark logic in building the URL path and redirect the Zeppelin user directly to a configured reverse proxy path
### What type of PR is it?
[Improvement]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2949
### How should this be tested?
* Set "zeppelin.spark.uiWebUrl" to http://localhost/spark-reverse-proxy-path
* Run Spark code
* Open Spark UI
* Expected to be redirected to http://localhost/spark-reverse-proxy-path
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? N
* Is there breaking changes for older versions? N
* Does this needs documentation? N
Author: Nelson Costa <ne...@gmail.com>
Closes #2596 from necosta/zeppelin2949 and squashes the following commits:
2ae8f2d [Nelson Costa] [ZEPPELIN-2949] Last amendments
118339f [Nelson Costa] [ZEPPELIN-2949] Updated documentation
826bad4 [Nelson Costa] [ZEPPELIN-2949] Updated documentation
0a56299 [Nelson Costa] [ZEPPELIN-2949] Allow custom Spark UI URL
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/c45128e7
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/c45128e7
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/c45128e7
Branch: refs/heads/master
Commit: c45128e79f619e6f2d927176a440ec44a2d588ed
Parents: 84cb4b5
Author: Nelson Costa <ne...@gmail.com>
Authored: Thu Oct 19 10:21:54 2017 +0100
Committer: Felix Cheung <fe...@apache.org>
Committed: Mon Oct 23 23:24:55 2017 -0700
----------------------------------------------------------------------
docs/interpreter/spark.md | 21 ++++++++++++--------
.../apache/zeppelin/spark/SparkInterpreter.java | 11 +++++-----
.../src/main/resources/interpreter-setting.json | 7 +++++++
.../zeppelin/spark/SparkInterpreterTest.java | 2 +-
4 files changed, 27 insertions(+), 14 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c45128e7/docs/interpreter/spark.md
----------------------------------------------------------------------
diff --git a/docs/interpreter/spark.md b/docs/interpreter/spark.md
index bbd9065..da957c6 100644
--- a/docs/interpreter/spark.md
+++ b/docs/interpreter/spark.md
@@ -145,6 +145,11 @@ You can also set other Spark properties which are not listed in the table. For a
<td>true</td>
<td>Do not change - developer only setting, not for production use</td>
</tr>
+ <tr>
+ <td>zeppelin.spark.uiWebUrl</td>
+ <td></td>
+ <td>Overrides Spark UI default URL. Value should be a full URL (ex: http://{hostName}/{uniquePath}</td>
+ </tr>
</table>
Without any configuration, Spark interpreter works out of box in local mode. But if you want to connect to your Spark cluster, you'll need to follow below two simple steps.
@@ -184,7 +189,7 @@ For example,
* **yarn-cluster** in Yarn cluster mode
* **mesos://host:5050** in Mesos cluster
-That's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding Zeppelin in this way.
+That's it. Zeppelin will work with any version of Spark and any deployment type without rebuilding Zeppelin in this way.
For the further information about Spark & Zeppelin version compatibility, please refer to "Available Interpreters" section in [Zeppelin download page](https://zeppelin.apache.org/download.html).
> Note that without exporting `SPARK_HOME`, it's running in local mode with included version of Spark. The included version may vary depending on the build profile.
@@ -216,7 +221,7 @@ There are two ways to load external libraries in Spark interpreter. First is usi
Please see [Dependency Management](../usage/interpreter/dependency_management.html) for the details.
### 2. Loading Spark Properties
-Once `SPARK_HOME` is set in `conf/zeppelin-env.sh`, Zeppelin uses `spark-submit` as spark interpreter runner. `spark-submit` supports two ways to load configurations.
+Once `SPARK_HOME` is set in `conf/zeppelin-env.sh`, Zeppelin uses `spark-submit` as spark interpreter runner. `spark-submit` supports two ways to load configurations.
The first is command line options such as --master and Zeppelin can pass these options to `spark-submit` by exporting `SPARK_SUBMIT_OPTIONS` in `conf/zeppelin-env.sh`. Second is reading configuration options from `SPARK_HOME/conf/spark-defaults.conf`. Spark properties that user can set to distribute libraries are:
<table class="table-configuration">
@@ -249,7 +254,7 @@ Here are few examples:
```bash
export SPARK_SUBMIT_OPTIONS="--packages com.databricks:spark-csv_2.10:1.2.0 --jars /path/mylib1.jar,/path/mylib2.jar --files /path/mylib1.py,/path/mylib2.zip,/path/mylib3.egg"
```
-
+
* `SPARK_HOME/conf/spark-defaults.conf`
```
@@ -414,17 +419,17 @@ To learn more about dynamic form, checkout [Dynamic Form](../usage/dynamic_form/
## Matplotlib Integration (pyspark)
-Both the `python` and `pyspark` interpreters have built-in support for inline visualization using `matplotlib`,
-a popular plotting library for python. More details can be found in the [python interpreter documentation](../interpreter/python.html),
-since matplotlib support is identical. More advanced interactive plotting can be done with pyspark through
+Both the `python` and `pyspark` interpreters have built-in support for inline visualization using `matplotlib`,
+a popular plotting library for python. More details can be found in the [python interpreter documentation](../interpreter/python.html),
+since matplotlib support is identical. More advanced interactive plotting can be done with pyspark through
utilizing Zeppelin's built-in [Angular Display System](../usage/display_system/angular_backend.html), as shown below:
<img class="img-responsive" src="{{BASE_PATH}}/assets/themes/zeppelin/img/docs-img/matplotlibAngularExample.gif" />
## Interpreter setting option
-You can choose one of `shared`, `scoped` and `isolated` options wheh you configure Spark interpreter.
-Spark interpreter creates separated Scala compiler per each notebook but share a single SparkContext in `scoped` mode (experimental).
+You can choose one of `shared`, `scoped` and `isolated` options wheh you configure Spark interpreter.
+Spark interpreter creates separated Scala compiler per each notebook but share a single SparkContext in `scoped` mode (experimental).
It creates separated SparkContext per each notebook in `isolated` mode.
## IPython support
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c45128e7/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
----------------------------------------------------------------------
diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
index 5a5110f..0536825 100644
--- a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
+++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java
@@ -26,11 +26,9 @@ import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;
-import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
import java.util.Map;
-import java.util.NoSuchElementException;
import java.util.Properties;
import java.util.Set;
import java.util.concurrent.atomic.AtomicInteger;
@@ -51,7 +49,6 @@ import org.apache.spark.sql.SQLContext;
import org.apache.spark.ui.SparkUI;
import org.apache.spark.ui.jobs.JobProgressListener;
import org.apache.zeppelin.interpreter.BaseZeppelinContext;
-import org.apache.zeppelin.interpreter.DefaultInterpreterProperty;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterException;
@@ -72,7 +69,6 @@ import org.apache.zeppelin.spark.dep.SparkDependencyResolver;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import com.google.common.base.Joiner;
import scala.Console;
import scala.Enumeration.Value;
import scala.None;
@@ -206,7 +202,7 @@ public class SparkInterpreter extends Interpreter {
private String getJobUrl(int jobId) {
String jobUrl = null;
if (sparkUrl != null) {
- jobUrl = sparkUrl + "/jobs/job?id=" + jobId;
+ jobUrl = sparkUrl + "/jobs/job/?id=" + jobId;
}
return jobUrl;
}
@@ -936,6 +932,11 @@ public class SparkInterpreter extends Interpreter {
return sparkUrl;
}
+ String sparkUrlProp = property.getProperty("zeppelin.spark.uiWebUrl", "");
+ if (!StringUtils.isBlank(sparkUrlProp)) {
+ return sparkUrlProp;
+ }
+
if (sparkVersion.newerThanEquals(SparkVersion.SPARK_2_0_0)) {
Option<String> uiWebUrlOption = (Option<String>) Utils.invokeMethod(sc, "uiWebUrl");
if (uiWebUrlOption.isDefined()) {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c45128e7/spark/src/main/resources/interpreter-setting.json
----------------------------------------------------------------------
diff --git a/spark/src/main/resources/interpreter-setting.json b/spark/src/main/resources/interpreter-setting.json
index 4aeda6c..485f695 100644
--- a/spark/src/main/resources/interpreter-setting.json
+++ b/spark/src/main/resources/interpreter-setting.json
@@ -67,6 +67,13 @@
"defaultValue": true,
"description": "Do not change - developer only setting, not for production use",
"type": "checkbox"
+ },
+ "zeppelin.spark.uiWebUrl": {
+ "envName": null,
+ "propertyName": "zeppelin.spark.uiWebUrl",
+ "defaultValue": "",
+ "description": "Override Spark UI default URL",
+ "type": "string"
}
},
"editor": {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c45128e7/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
----------------------------------------------------------------------
diff --git a/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java b/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
index aaf13c3..10141a1 100644
--- a/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
+++ b/spark/src/test/java/org/apache/zeppelin/spark/SparkInterpreterTest.java
@@ -347,7 +347,7 @@ public class SparkInterpreterTest {
}
String sparkUIUrl = repl.getSparkUIUrl();
assertNotNull(jobUrl);
- assertTrue(jobUrl.startsWith(sparkUIUrl + "/jobs/job?id="));
+ assertTrue(jobUrl.startsWith(sparkUIUrl + "/jobs/job/?id="));
}
}