You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by pd...@apache.org on 2021/09/02 14:27:37 UTC
[zeppelin] branch master updated: [ZEPPELIN-5445] Allow users to
access the spark UI outside the kubernetes cluster
This is an automated email from the ASF dual-hosted git repository.
pdallig pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new ba76694 [ZEPPELIN-5445] Allow users to access the spark UI outside the kubernetes cluster
ba76694 is described below
commit ba766944b96308f117d488803a0d3a529b488dc6
Author: rick <ri...@rickdeMacBook-Pro.local>
AuthorDate: Tue Jul 20 18:31:15 2021 +0800
[ZEPPELIN-5445] Allow users to access the spark UI outside the kubernetes cluster
### What is this PR for?
Currently, when zeppelin server is running outside of k8s cluster, users cannot access the spark UI directly from the web page of Zeppelin. So this PR creates ingress resource in `k8s/interpreter/100-interpreter-spec.yaml` and enables the `SPARK JOB` button to jump directly to the URL defined by ingress.
If user does not specify the property `zeppelin.spark.uiWebUrl`, it will be automatically generated according to the template: `{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}`, which is also the URL that the `SPARK JOB` button points to. So this PR also configured the ingress.host by this URL.
This PR also adds a property `zeppelin.k8s.spark.useIngress` to let user choose whether to create ingress.
Example usage:
```
zeppelin.k8s.spark.useIngress true
```
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* <https://issues.apache.org/jira/browse/ZEPPELIN-5445>
### How should this be tested?
* CI pass and manually tested
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: rick <ri...@rickdeMacBook-Pro.local>
Closes #4176 from rickchengx/ZEPPELIN-5445 and squashes the following commits:
fdfe89ad3 [rick] update docs
1773d0657 [rick] [ZEPPELIN-5445] Allow users to access the spark UI outside the kubernetes cluster
---
docs/quickstart/kubernetes.md | 4 +++
k8s/interpreter/100-interpreter-spec.yaml | 30 ++++++++++++++++++++++
.../launcher/K8sRemoteInterpreterProcess.java | 14 ++++++++++
3 files changed, 48 insertions(+)
diff --git a/docs/quickstart/kubernetes.md b/docs/quickstart/kubernetes.md
index 1c0b99a..7c7f88a 100644
--- a/docs/quickstart/kubernetes.md
+++ b/docs/quickstart/kubernetes.md
@@ -245,6 +245,7 @@ to customize,
4. Run a paragraph will create an interpreter using modified yaml files.
The interpreter pod can also be customized through the interpreter settings. Here are some of the properties:
+
| Property Name | Default Value | Description |
| ----- | ----- | ----- |
| `zeppelin.k8s.namespace` | `default` | The Kubernetes namespace to use. |
@@ -256,6 +257,9 @@ The interpreter pod can also be customized through the interpreter settings. Her
| `zeppelin.k8s.interpreter.imagePullSecrets` | (optional) | Set the comma-separated list of Kubernetes secrets while pulling images, e.g., `mysecret1,mysecret2` |
| `zeppelin.k8s.interpreter.container.imagePullPolicy` | (optional) | Set the pull policy of the interpreter image, e.g., `Always` |
| `zeppelin.k8s.spark.container.imagePullPolicy` | (optional) | Set the pull policy of the spark image, e.g., `Always` |
+| `zeppelin.spark.uiWebUrl` | `//{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}` | The URL for user to access Spark UI. The default value is a [jinjava](https://github.com/HubSpot/jinjava) template that contains three variables. |
+| `zeppelin.k8s.spark.useIngress` | (optional) | If true, the [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) will be created when creating the spark interpreter. So users can access the Spark UI through Ingress. |
+| `zeppelin.k8s.spark.ingress.host` | `{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}` | If `zeppelin.k8s.spark.useIngress` is `true`, it configures the `host` value of the Ingress. The default value is a [jinjava](https://github.com/HubSpot/jinjava) template that contains three variables. Users can access the Spark UI through a customized `zeppelin.k8s.spark.ingress.host`. |
## Future work
diff --git a/k8s/interpreter/100-interpreter-spec.yaml b/k8s/interpreter/100-interpreter-spec.yaml
index 1bf1615..91d6691 100644
--- a/k8s/interpreter/100-interpreter-spec.yaml
+++ b/k8s/interpreter/100-interpreter-spec.yaml
@@ -180,4 +180,34 @@ roleRef:
kind: Role
name: {{zeppelin.k8s.interpreter.pod.name}}
apiGroup: rbac.authorization.k8s.io
+{% if zeppelin.k8s.spark.useIngress is defined and zeppelin.k8s.spark.useIngress == "true" %}
+---
+# create ingress of spark UI
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: spark-ui-{{zeppelin.k8s.interpreter.pod.name}}
+ namespace: {{zeppelin.k8s.namespace}}
+ {% if zeppelin.k8s.server.uid is defined %}
+ ownerReferences:
+ - apiVersion: v1
+ controller: false
+ blockOwnerDeletion: false
+ kind: Pod
+ name: {{zeppelin.k8s.server.pod.name}}
+ uid: {{zeppelin.k8s.server.uid}}
+ {% endif %}
+spec:
+ rules:
+ - host: {{zeppelin.k8s.spark.ingress.host}}
+ http:
+ paths:
+ - pathType: Prefix
+ path: "/"
+ backend:
+ service:
+ name: {{zeppelin.k8s.interpreter.pod.name}}
+ port:
+ number: 4040
+{% endif %}
{% endif %}
\ No newline at end of file
diff --git a/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java b/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
index bfd016f..c047ed7 100644
--- a/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
+++ b/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
@@ -318,6 +318,20 @@ public class K8sRemoteInterpreterProcess extends RemoteInterpreterManagedProcess
getPodName(),
k8sEnv.get(ENV_SERVICE_DOMAIN)
));
+
+ // configure interpreter property "zeppelin.k8s.spark.ingress.host" if not defined, to enable spark ui through ingress
+ String ingressHost = (String) properties.get("zeppelin.k8s.spark.ingress.host");
+ if (StringUtils.isBlank(ingressHost)) {
+ ingressHost = "{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}";
+ }
+ properties.put("zeppelin.k8s.spark.ingress.host",
+ sparkUiWebUrlFromTemplate(
+ ingressHost,
+ webUiPort,
+ getPodName(),
+ k8sEnv.get(ENV_SERVICE_DOMAIN)
+ ));
+
// Resources of Interpreter Pod
if (properties.containsKey(SPARK_DRIVER_MEMORY)) {
String memory;