You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by li...@apache.org on 2021/02/05 14:06:17 UTC
[incubator-dolphinscheduler] branch 1.3.5-prepare updated:
[1.3.5-prepare][Improvement][K8s] Improve k8s for helm and update readme
(#4697)
This is an automated email from the ASF dual-hosted git repository.
lidongdai pushed a commit to branch 1.3.5-prepare
in repository https://gitbox.apache.org/repos/asf/incubator-dolphinscheduler.git
The following commit(s) were added to refs/heads/1.3.5-prepare by this push:
new 223fe99 [1.3.5-prepare][Improvement][K8s] Improve k8s for helm and update readme (#4697)
223fe99 is described below
commit 223fe993b72893cd817fda7fc3d5723f9cc4adc4
Author: Shiwen Cheng <ch...@gmail.com>
AuthorDate: Fri Feb 5 22:06:07 2021 +0800
[1.3.5-prepare][Improvement][K8s] Improve k8s for helm and update readme (#4697)
---
docker/kubernetes/dolphinscheduler/Chart.yaml | 12 +--
docker/kubernetes/dolphinscheduler/README.md | 58 +++++------
.../kubernetes/dolphinscheduler/requirements.yaml | 25 -----
.../dolphinscheduler/templates/NOTES.txt | 19 +---
.../configmap-dolphinscheduler-master.yaml | 1 -
...ml => configmap-dolphinscheduler-resource.yaml} | 23 ++---
.../configmap-dolphinscheduler-worker.yaml | 1 -
.../deployment-dolphinscheduler-alert.yaml | 39 +-------
.../templates/deployment-dolphinscheduler-api.yaml | 77 ++++++++-------
.../deployment-dolphinscheduler-frontend.yaml | 102 -------------------
.../dolphinscheduler/templates/ingress.yaml | 2 +-
.../templates/pvc-dolphinscheduler-alert.yaml | 2 +-
.../templates/pvc-dolphinscheduler-api.yaml | 2 +-
...duler-frontend.yaml => secret-resource-s3.yaml} | 22 ++---
.../statefulset-dolphinscheduler-master.yaml | 74 +++-----------
.../statefulset-dolphinscheduler-worker.yaml | 110 +++++++++------------
.../svc-dolphinscheduler-worker-headless.yaml | 4 +-
docker/kubernetes/dolphinscheduler/values.yaml | 106 +++++++-------------
18 files changed, 193 insertions(+), 486 deletions(-)
diff --git a/docker/kubernetes/dolphinscheduler/Chart.yaml b/docker/kubernetes/dolphinscheduler/Chart.yaml
index ac989d5..baea43a 100644
--- a/docker/kubernetes/dolphinscheduler/Chart.yaml
+++ b/docker/kubernetes/dolphinscheduler/Chart.yaml
@@ -22,7 +22,7 @@ home: https://dolphinscheduler.apache.org
icon: https://dolphinscheduler.apache.org/img/hlogo_colorful.svg
keywords:
- dolphinscheduler
-- Scheduler
+- scheduler
# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
@@ -35,18 +35,18 @@ type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
-version: 1.0.0
+version: 1.1.0
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
-appVersion: 1.3.0
+appVersion: 1.3.5
dependencies:
- name: postgresql
- version: 8.x.x
+ version: 10.x.x
repository: https://charts.bitnami.com/bitnami
condition: postgresql.enabled
- name: zookeeper
- version: 5.x.x
+ version: 6.x.x
repository: https://charts.bitnami.com/bitnami
- condition: redis.enabled
+ condition: zookeeper.enabled
diff --git a/docker/kubernetes/dolphinscheduler/README.md b/docker/kubernetes/dolphinscheduler/README.md
index 9e6d1c6..cf69346 100644
--- a/docker/kubernetes/dolphinscheduler/README.md
+++ b/docker/kubernetes/dolphinscheduler/README.md
@@ -7,19 +7,20 @@ This chart bootstraps a [Dolphin Scheduler](https://dolphinscheduler.apache.org)
## Prerequisites
-- Kubernetes 1.10+
+- Helm 3.1.0+
+- Kubernetes 1.12+
- PV provisioner support in the underlying infrastructure
## Installing the Chart
-To install the chart with the release name `my-release`:
+To install the chart with the release name `dolphinscheduler`:
```bash
$ git clone https://github.com/apache/incubator-dolphinscheduler.git
-$ cd incubator-dolphinscheduler/kubernetes/dolphinscheduler
+$ cd incubator-dolphinscheduler/docker/kubernetes/dolphinscheduler
$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm dependency update .
-$ helm install --name dolphinscheduler .
+$ helm install dolphinscheduler .
```
These commands deploy Dolphin Scheduler on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation.
@@ -30,7 +31,7 @@ These commands deploy Dolphin Scheduler on the Kubernetes cluster in the default
To uninstall/delete the `dolphinscheduler` deployment:
```bash
-$ helm delete --purge dolphinscheduler
+$ helm uninstall dolphinscheduler
```
The command removes all the Kubernetes components associated with the chart and deletes the release.
@@ -41,12 +42,14 @@ The following tables lists the configurable parameters of the Dolphins Scheduler
| Parameter | Description | Default |
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------- |
+| `nameOverride` | String to partially override common.names.fullname | `nil` |
+| `fullnameOverride` | String to fully override common.names.fullname | `nil` |
| `timezone` | World time and date for cities in all time zones | `Asia/Shanghai` |
| `image.registry` | Docker image registry for the Dolphins Scheduler | `docker.io` |
| `image.repository` | Docker image repository for the Dolphins Scheduler | `dolphinscheduler` |
| `image.tag` | Docker image version for the Dolphins Scheduler | `1.2.1` |
| `image.imagePullPolicy` | Image pull policy. One of Always, Never, IfNotPresent | `IfNotPresent` |
-| `imagePullSecrets` | ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images | `[]` |
+| `imagePullSecrets` | ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images | `[]` |
| | | |
| `postgresql.enabled` | If not exists external PostgreSQL, by default, the Dolphins Scheduler will use a internal PostgreSQL | `true` |
| `postgresql.postgresqlUsername` | The username for internal PostgreSQL | `root` |
@@ -55,19 +58,28 @@ The following tables lists the configurable parameters of the Dolphins Scheduler
| `postgresql.persistence.enabled` | Set `postgresql.persistence.enabled` to `true` to mount a new volume for internal PostgreSQL | `false` |
| `postgresql.persistence.size` | `PersistentVolumeClaim` Size | `20Gi` |
| `postgresql.persistence.storageClass` | PostgreSQL data Persistent Volume Storage Class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
-| `externalDatabase.host` | If exists external PostgreSQL, and set `postgresql.enable` value to false. Dolphins Scheduler's database host will use it. | `localhost` |
+| `externalDatabase.host` | If exists external PostgreSQL, and set `postgresql.enable` value to false. Dolphins Scheduler's database host will use it. | `localhost` |
| `externalDatabase.port` | If exists external PostgreSQL, and set `postgresql.enable` value to false. Dolphins Scheduler's database port will use it. | `5432` |
| `externalDatabase.username` | If exists external PostgreSQL, and set `postgresql.enable` value to false. Dolphins Scheduler's database username will use it. | `root` |
| `externalDatabase.password` | If exists external PostgreSQL, and set `postgresql.enable` value to false. Dolphins Scheduler's database password will use it. | `root` |
| `externalDatabase.database` | If exists external PostgreSQL, and set `postgresql.enable` value to false. Dolphins Scheduler's database database will use it. | `dolphinscheduler` |
| | | |
| `zookeeper.enabled` | If not exists external Zookeeper, by default, the Dolphin Scheduler will use a internal Zookeeper | `true` |
-| `zookeeper.taskQueue` | Specify task queue for `master` and `worker` | `zookeeper` |
+| `zookeeper.fourlwCommandsWhitelist` | A list of comma separated Four Letter Words commands to use | `srvr,ruok,wchs,cons` |
+| `zookeeper.service.port` | ZooKeeper port | `2181` |
| `zookeeper.persistence.enabled` | Set `zookeeper.persistence.enabled` to `true` to mount a new volume for internal Zookeeper | `false` |
| `zookeeper.persistence.size` | `PersistentVolumeClaim` Size | `20Gi` |
| `zookeeper.persistence.storageClass` | Zookeeper data Persistent Volume Storage Class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
-| `externalZookeeper.taskQueue` | If exists external Zookeeper, and set `zookeeper.enable` value to false. Specify task queue for `master` and `worker` | `zookeeper` |
+| `zookeeper.zookeeperRoot` | Specify dolphinscheduler root directory in Zookeeper | `/dolphinscheduler` |
| `externalZookeeper.zookeeperQuorum` | If exists external Zookeeper, and set `zookeeper.enable` value to false. Specify Zookeeper quorum | `127.0.0.1:2181` |
+| `externalZookeeper.zookeeperRoot` | If exists external Zookeeper, and set `zookeeper.enable` value to false. Specify dolphinscheduler root directory in Zookeeper | `/dolphinscheduler` |
+| | | |
+| `resource.resourceStorageType` | Resource storage type: HDFS, S3, NONE | `HDFS` |
+| `resource.resourceUploadPath` | Resource store on HDFS/S3 path, please make sure the directory exists on hdfs and have read write permissions | `/dolphinscheduler` |
+| `resource.fsDefaultFS` | Resource storage file system like `file:///`, `hdfs://mycluster:8020` or `s3a://dolphinscheduler` | `file:///` |
+| `resource.fsS3aEndpoint` | S3 endpoint when `resource.resourceStorageType` be seted to `S3` | `s3.xxx.amazonaws.com` |
+| `resource.fsS3aAccessKey` | S3 access key when `resource.resourceStorageType` be seted to `S3` | `xxxxxxx` |
+| `resource.fsS3aSecretKey` | S3 secret key when `resource.resourceStorageType` be seted to `S3` | `xxxxxxx` |
| | | |
| `master.podManagementPolicy` | PodManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down | `Parallel` |
| `master.replicas` | Replicas is the desired number of replicas of the given Template | `3` |
@@ -81,6 +93,7 @@ The following tables lists the configurable parameters of the Dolphins Scheduler
| `master.configmap.MASTER_TASK_COMMIT_INTERVAL` | Master commit task interval | `1000` |
| `master.configmap.MASTER_MAX_CPULOAD_AVG` | Only less than cpu avg load, master server can work. default value : the number of cpu cores * 2 | `100` |
| `master.configmap.MASTER_RESERVED_MEMORY` | Only larger than reserved memory, master server can work. default value : physical memory * 1/10, unit is G | `0.1` |
+| `master.configmap.MASTER_LISTEN_PORT` | Master listen port | `5678` |
| `master.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
| `master.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
| `master.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
@@ -105,9 +118,10 @@ The following tables lists the configurable parameters of the Dolphins Scheduler
| `worker.affinity` | If specified, the pod's scheduling constraints | `{}` |
| `worker.configmap.WORKER_EXEC_THREADS` | Worker execute thread num | `100` |
| `worker.configmap.WORKER_HEARTBEAT_INTERVAL` | Worker heartbeat interval | `10` |
-| `worker.configmap.WORKER_FETCH_TASK_NUM` | Submit the number of tasks at a time | `3` |
| `worker.configmap.WORKER_MAX_CPULOAD_AVG` | Only less than cpu avg load, worker server can work. default value : the number of cpu cores * 2 | `100` |
| `worker.configmap.WORKER_RESERVED_MEMORY` | Only larger than reserved memory, worker server can work. default value : physical memory * 1/10, unit is G | `0.1` |
+| `worker.configmap.WORKER_LISTEN_PORT` | Worker listen port | `1234` |
+| `worker.configmap.WORKER_GROUP` | Worker group | `default` |
| `worker.configmap.DOLPHINSCHEDULER_DATA_BASEDIR_PATH` | User data directory path, self configuration, please make sure the directory exists and have read write permissions | `/tmp/dolphinscheduler` |
| `worker.configmap.DOLPHINSCHEDULER_ENV` | System env path, self configuration, please read `values.yaml` | `[]` |
| `worker.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
@@ -194,30 +208,6 @@ The following tables lists the configurable parameters of the Dolphins Scheduler
| `api.persistentVolumeClaim.storageClassName` | `api` logs data Persistent Volume Storage Class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `api.persistentVolumeClaim.storage` | `PersistentVolumeClaim` Size | `20Gi` |
| | | |
-| `frontend.strategy.type` | Type of deployment. Can be "Recreate" or "RollingUpdate" | `RollingUpdate` |
-| `frontend.strategy.rollingUpdate.maxSurge` | The maximum number of pods that can be scheduled above the desired number of pods | `25%` |
-| `frontend.strategy.rollingUpdate.maxUnavailable` | The maximum number of pods that can be unavailable during the update | `25%` |
-| `frontend.replicas` | Replicas is the desired number of replicas of the given Template | `1` |
-| `frontend.nodeSelector` | NodeSelector is a selector which must be true for the pod to fit on a node | `{}` |
-| `frontend.tolerations` | If specified, the pod's tolerations | `{}` |
-| `frontend.affinity` | If specified, the pod's scheduling constraints | `{}` |
-| `frontend.livenessProbe.enabled` | Turn on and off liveness probe | `true` |
-| `frontend.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `30` |
-| `frontend.livenessProbe.periodSeconds` | How often to perform the probe | `30` |
-| `frontend.livenessProbe.timeoutSeconds` | When the probe times out | `5` |
-| `frontend.livenessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
-| `frontend.livenessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
-| `frontend.readinessProbe.enabled` | Turn on and off readiness probe | `true` |
-| `frontend.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` |
-| `frontend.readinessProbe.periodSeconds` | How often to perform the probe | `30` |
-| `frontend.readinessProbe.timeoutSeconds` | When the probe times out | `5` |
-| `frontend.readinessProbe.failureThreshold` | Minimum consecutive successes for the probe | `3` |
-| `frontend.readinessProbe.successThreshold` | Minimum consecutive failures for the probe | `1` |
-| `frontend.persistentVolumeClaim.enabled` | Set `frontend.persistentVolumeClaim.enabled` to `true` to mount a new volume for `frontend` | `false` |
-| `frontend.persistentVolumeClaim.accessModes` | `PersistentVolumeClaim` Access Modes | `[ReadWriteOnce]` |
-| `frontend.persistentVolumeClaim.storageClassName` | `frontend` logs data Persistent Volume Storage Class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
-| `frontend.persistentVolumeClaim.storage` | `PersistentVolumeClaim` Size | `20Gi` |
-| | | |
| `ingress.enabled` | Enable ingress | `false` |
| `ingress.host` | Ingress host | `dolphinscheduler.org` |
| `ingress.path` | Ingress path | `/` |
diff --git a/docker/kubernetes/dolphinscheduler/requirements.yaml b/docker/kubernetes/dolphinscheduler/requirements.yaml
deleted file mode 100644
index e219975..0000000
--- a/docker/kubernetes/dolphinscheduler/requirements.yaml
+++ /dev/null
@@ -1,25 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-dependencies:
-- name: postgresql
- version: 8.x.x
- repository: https://charts.bitnami.com/bitnami
- condition: postgresql.enabled
-- name: zookeeper
- version: 5.x.x
- repository: https://charts.bitnami.com/bitnami
- condition: redis.enabled
\ No newline at end of file
diff --git a/docker/kubernetes/dolphinscheduler/templates/NOTES.txt b/docker/kubernetes/dolphinscheduler/templates/NOTES.txt
index eb3a9cf..717abee 100644
--- a/docker/kubernetes/dolphinscheduler/templates/NOTES.txt
+++ b/docker/kubernetes/dolphinscheduler/templates/NOTES.txt
@@ -15,9 +15,9 @@
# limitations under the License.
#
-** Please be patient while the chart is being deployed **
+** Please be patient while the chart Dolphinscheduler {{ .Chart.AppVersion }} is being deployed **
-1. Get the Dolphinscheduler URL by running:
+Get the Dolphinscheduler URL by running:
{{- if .Values.ingress.enabled }}
@@ -26,19 +26,6 @@
{{- else }}
- kubectl port-forward --namespace {{ .Release.Namespace }} svc/{{ template "dolphinscheduler.fullname" . }}-frontend 8888:8888
-
-{{- end }}
-
-2. Get the Dolphinscheduler URL by running:
-
-{{- if .Values.ingress.enabled }}
-
- export HOSTNAME=$(kubectl get ingress --namespace {{ .Release.Namespace }} {{ template "dolphinscheduler.fullname" . }} -o jsonpath='{.spec.rules[0].host}')
- echo "Dolphinscheduler URL: http://$HOSTNAME/"
-
-{{- else }}
-
- kubectl port-forward --namespace {{ .Release.Namespace }} svc/{{ template "dolphinscheduler.fullname" . }}-frontend 8888:8888
+ kubectl port-forward --namespace {{ .Release.Namespace }} svc/{{ template "dolphinscheduler.fullname" . }}-api 12345:12345
{{- end }}
\ No newline at end of file
diff --git a/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-master.yaml b/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-master.yaml
index da82d63..fa7b7dd 100644
--- a/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-master.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-master.yaml
@@ -32,5 +32,4 @@ data:
MASTER_MAX_CPULOAD_AVG: {{ .Values.master.configmap.MASTER_MAX_CPULOAD_AVG | quote }}
MASTER_RESERVED_MEMORY: {{ .Values.master.configmap.MASTER_RESERVED_MEMORY | quote }}
MASTER_LISTEN_PORT: {{ .Values.master.configmap.MASTER_LISTEN_PORT | quote }}
- DOLPHINSCHEDULER_DATA_BASEDIR_PATH: {{ include "dolphinscheduler.worker.base.dir" . | quote }}
{{- end }}
\ No newline at end of file
diff --git a/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-frontend.yaml b/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-resource.yaml
similarity index 67%
rename from docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-frontend.yaml
rename to docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-resource.yaml
index ac9fe02..f8b0522 100644
--- a/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-frontend.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-resource.yaml
@@ -14,22 +14,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
-{{- if .Values.frontend.persistentVolumeClaim.enabled }}
+{{- if .Values.resource }}
apiVersion: v1
-kind: PersistentVolumeClaim
+kind: ConfigMap
metadata:
- name: {{ include "dolphinscheduler.fullname" . }}-frontend
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
labels:
- app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-frontend
+ app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-resource
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
-spec:
- accessModes:
- {{- range .Values.frontend.persistentVolumeClaim.accessModes }}
- - {{ . | quote }}
- {{- end }}
- storageClassName: {{ .Values.frontend.persistentVolumeClaim.storageClassName | quote }}
- resources:
- requests:
- storage: {{ .Values.frontend.persistentVolumeClaim.storage | quote }}
+data:
+ RESOURCE_STORAGE_TYPE: {{ .Values.resource.resourceStorageType | quote }}
+ RESOURCE_UPLOAD_PATH: {{ .Values.resource.resourceUploadPath | quote }}
+ FS_DEFAULT_FS: {{ .Values.resource.fsDefaultFS | quote }}
+ FS_S3A_ENDPOINT: {{ .Values.resource.fsS3aEndpoint | quote }}
+ FS_S3A_ACCESS_KEY: {{ .Values.resource.fsS3aAccessKey | quote }}
{{- end }}
\ No newline at end of file
diff --git a/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-worker.yaml b/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-worker.yaml
index 1e08b67..2aa1eab 100644
--- a/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-worker.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/configmap-dolphinscheduler-worker.yaml
@@ -26,7 +26,6 @@ metadata:
data:
WORKER_EXEC_THREADS: {{ .Values.worker.configmap.WORKER_EXEC_THREADS | quote }}
WORKER_HEARTBEAT_INTERVAL: {{ .Values.worker.configmap.WORKER_HEARTBEAT_INTERVAL | quote }}
- WORKER_FETCH_TASK_NUM: {{ .Values.worker.configmap.WORKER_FETCH_TASK_NUM | quote }}
WORKER_MAX_CPULOAD_AVG: {{ .Values.worker.configmap.WORKER_MAX_CPULOAD_AVG | quote }}
WORKER_RESERVED_MEMORY: {{ .Values.worker.configmap.WORKER_RESERVED_MEMORY | quote }}
WORKER_LISTEN_PORT: {{ .Values.worker.configmap.WORKER_LISTEN_PORT | quote }}
diff --git a/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml b/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml
index 69662e9..5e45218 100644
--- a/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-alert.yaml
@@ -53,35 +53,6 @@ spec:
{{- if .Values.alert.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
- initContainers:
- - name: init-postgresql
- image: busybox:1.31.0
- command:
- - /bin/sh
- - -ec
- - |
- while ! nc -z ${POSTGRESQL_HOST} ${POSTGRESQL_PORT}; do
- counter=$((counter+1))
- if [ $counter == 5 ]; then
- echo "Error: Couldn't connect to postgresql."
- exit 1
- fi
- echo "Trying to connect to postgresql at ${POSTGRESQL_HOST}:${POSTGRESQL_PORT}. Attempt $counter."
- sleep 60
- done
- env:
- - name: POSTGRESQL_HOST
- {{- if .Values.postgresql.enabled }}
- value: {{ template "dolphinscheduler.postgresql.fullname" . }}
- {{- else }}
- value: {{ .Values.externalDatabase.host | quote }}
- {{- end }}
- - name: POSTGRESQL_PORT
- {{- if .Values.postgresql.enabled }}
- value: "5432"
- {{- else }}
- value: {{ .Values.externalDatabase.port }}
- {{- end }}
containers:
- name: {{ include "dolphinscheduler.fullname" . }}-alert
image: {{ include "dolphinscheduler.image.repository" . | quote }}
@@ -161,25 +132,25 @@ spec:
configMapKeyRef:
key: ENTERPRISE_WECHAT_USERS
name: {{ include "dolphinscheduler.fullname" . }}-alert
- - name: POSTGRESQL_HOST
+ - name: DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "dolphinscheduler.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- - name: POSTGRESQL_PORT
+ - name: DATABASE_PORT
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port }}
{{- end }}
- - name: POSTGRESQL_USERNAME
+ - name: DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.username | quote }}
{{- end }}
- - name: POSTGRESQL_PASSWORD
+ - name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.postgresql.enabled }}
@@ -189,7 +160,7 @@ spec:
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
{{- end }}
- - name: POSTGRESQL_DATABASE
+ - name: DATABASE_DATABASE
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
diff --git a/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml b/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml
index 487ede0..0db3d55 100644
--- a/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-api.yaml
@@ -53,35 +53,6 @@ spec:
{{- if .Values.api.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
- initContainers:
- - name: init-postgresql
- image: busybox:1.31.0
- command:
- - /bin/sh
- - -ec
- - |
- while ! nc -z ${POSTGRESQL_HOST} ${POSTGRESQL_PORT}; do
- counter=$((counter+1))
- if [ $counter == 5 ]; then
- echo "Error: Couldn't connect to postgresql."
- exit 1
- fi
- echo "Trying to connect to postgresql at ${POSTGRESQL_HOST}:${POSTGRESQL_PORT}. Attempt $counter."
- sleep 60
- done
- env:
- - name: POSTGRESQL_HOST
- {{- if .Values.postgresql.enabled }}
- value: {{ template "dolphinscheduler.postgresql.fullname" . }}
- {{- else }}
- value: {{ .Values.externalDatabase.host | quote }}
- {{- end }}
- - name: POSTGRESQL_PORT
- {{- if .Values.postgresql.enabled }}
- value: "5432"
- {{- else }}
- value: {{ .Values.externalDatabase.port }}
- {{- end }}
containers:
- name: {{ include "dolphinscheduler.fullname" . }}-api
image: {{ include "dolphinscheduler.image.repository" . | quote }}
@@ -94,25 +65,25 @@ spec:
env:
- name: TZ
value: {{ .Values.timezone }}
- - name: POSTGRESQL_HOST
+ - name: DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "dolphinscheduler.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- - name: POSTGRESQL_PORT
+ - name: DATABASE_PORT
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port }}
{{- end }}
- - name: POSTGRESQL_USERNAME
+ - name: DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.username | quote }}
{{- end }}
- - name: POSTGRESQL_PASSWORD
+ - name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.postgresql.enabled }}
@@ -122,7 +93,7 @@ spec:
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
{{- end }}
- - name: POSTGRESQL_DATABASE
+ - name: DATABASE_DATABASE
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
@@ -134,6 +105,44 @@ spec:
{{- else }}
value: {{ .Values.externalZookeeper.zookeeperQuorum }}
{{- end }}
+ - name: ZOOKEEPER_ROOT
+ {{- if .Values.zookeeper.enabled }}
+ value: {{ .Values.zookeeper.zookeeperRoot }}
+ {{- else }}
+ value: {{ .Values.externalZookeeper.zookeeperRoot }}
+ {{- end }}
+ - name: RESOURCE_STORAGE_TYPE
+ valueFrom:
+ configMapKeyRef:
+ key: RESOURCE_STORAGE_TYPE
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ - name: RESOURCE_UPLOAD_PATH
+ valueFrom:
+ configMapKeyRef:
+ key: RESOURCE_UPLOAD_PATH
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ - name: FS_DEFAULT_FS
+ valueFrom:
+ configMapKeyRef:
+ key: FS_DEFAULT_FS
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ {{- if eq .Values.resource.resourceStorageType "S3" }}
+ - name: FS_S3A_ENDPOINT
+ valueFrom:
+ configMapKeyRef:
+ key: FS_S3A_ENDPOINT
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ - name: FS_S3A_ACCESS_KEY
+ valueFrom:
+ configMapKeyRef:
+ key: FS_S3A_ACCESS_KEY
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ - name: FS_S3A_SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ key: fs-s3a-secret-key
+ name: {{ printf "%s-%s" .Release.Name "resource-s3" }}
+ {{- end }}
{{- if .Values.api.livenessProbe.enabled }}
livenessProbe:
tcpSocket:
diff --git a/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-frontend.yaml b/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-frontend.yaml
deleted file mode 100644
index aea09f1..0000000
--- a/docker/kubernetes/dolphinscheduler/templates/deployment-dolphinscheduler-frontend.yaml
+++ /dev/null
@@ -1,102 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: {{ include "dolphinscheduler.fullname" . }}-frontend
- labels:
- app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-frontend
- app.kubernetes.io/instance: {{ .Release.Name }}
- app.kubernetes.io/managed-by: {{ .Release.Service }}
- app.kubernetes.io/component: frontend
-spec:
- replicas: {{ .Values.frontend.replicas }}
- selector:
- matchLabels:
- app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-frontend
- app.kubernetes.io/instance: {{ .Release.Name }}
- app.kubernetes.io/managed-by: {{ .Release.Service }}
- app.kubernetes.io/component: frontend
- strategy:
- type: {{ .Values.frontend.strategy.type | quote }}
- rollingUpdate:
- maxSurge: {{ .Values.frontend.strategy.rollingUpdate.maxSurge | quote }}
- maxUnavailable: {{ .Values.frontend.strategy.rollingUpdate.maxUnavailable | quote }}
- template:
- metadata:
- labels:
- app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-frontend
- app.kubernetes.io/instance: {{ .Release.Name }}
- app.kubernetes.io/managed-by: {{ .Release.Service }}
- app.kubernetes.io/component: frontend
- spec:
- {{- if .Values.frontend.affinity }}
- affinity: {{- toYaml .Values.frontend.affinity | nindent 8 }}
- {{- end }}
- {{- if .Values.frontend.nodeSelector }}
- nodeSelector: {{- toYaml .Values.frontend.nodeSelector | nindent 8 }}
- {{- end }}
- {{- if .Values.frontend.tolerations }}
- tolerations: {{- toYaml . | nindent 8 }}
- {{- end }}
- containers:
- - name: {{ include "dolphinscheduler.fullname" . }}-frontend
- image: {{ include "dolphinscheduler.image.repository" . | quote }}
- args:
- - "frontend"
- ports:
- - containerPort: 8888
- name: tcp-port
- imagePullPolicy: {{ .Values.image.pullPolicy }}
- env:
- - name: TZ
- value: {{ .Values.timezone }}
- - name: FRONTEND_API_SERVER_HOST
- value: '{{ include "dolphinscheduler.fullname" . }}-api'
- - name: FRONTEND_API_SERVER_PORT
- value: "12345"
- {{- if .Values.frontend.livenessProbe.enabled }}
- livenessProbe:
- tcpSocket:
- port: 8888
- initialDelaySeconds: {{ .Values.frontend.livenessProbe.initialDelaySeconds }}
- periodSeconds: {{ .Values.frontend.livenessProbe.periodSeconds }}
- timeoutSeconds: {{ .Values.frontend.livenessProbe.timeoutSeconds }}
- successThreshold: {{ .Values.frontend.livenessProbe.successThreshold }}
- failureThreshold: {{ .Values.frontend.livenessProbe.failureThreshold }}
- {{- end }}
- {{- if .Values.frontend.readinessProbe.enabled }}
- readinessProbe:
- tcpSocket:
- port: 8888
- initialDelaySeconds: {{ .Values.frontend.readinessProbe.initialDelaySeconds }}
- periodSeconds: {{ .Values.frontend.readinessProbe.periodSeconds }}
- timeoutSeconds: {{ .Values.frontend.readinessProbe.timeoutSeconds }}
- successThreshold: {{ .Values.frontend.readinessProbe.successThreshold }}
- failureThreshold: {{ .Values.frontend.readinessProbe.failureThreshold }}
- {{- end }}
- volumeMounts:
- - mountPath: "/var/log/nginx"
- name: {{ include "dolphinscheduler.fullname" . }}-frontend
- volumes:
- - name: {{ include "dolphinscheduler.fullname" . }}-frontend
- {{- if .Values.frontend.persistentVolumeClaim.enabled }}
- persistentVolumeClaim:
- claimName: {{ include "dolphinscheduler.fullname" . }}-frontend
- {{- else }}
- emptyDir: {}
- {{- end }}
\ No newline at end of file
diff --git a/docker/kubernetes/dolphinscheduler/templates/ingress.yaml b/docker/kubernetes/dolphinscheduler/templates/ingress.yaml
index d0f923d..6edc82b 100644
--- a/docker/kubernetes/dolphinscheduler/templates/ingress.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/ingress.yaml
@@ -30,7 +30,7 @@ spec:
paths:
- path: {{ .Values.ingress.path }}
backend:
- serviceName: {{ include "dolphinscheduler.fullname" . }}-frontend
+ serviceName: {{ include "dolphinscheduler.fullname" . }}-api
servicePort: tcp-port
{{- if .Values.ingress.tls.enabled }}
tls:
diff --git a/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-alert.yaml b/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-alert.yaml
index 7f74cd9..fd5e88a 100644
--- a/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-alert.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-alert.yaml
@@ -25,7 +25,7 @@ metadata:
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
accessModes:
- {{- range .Values.alert.persistentVolumeClaim.accessModes }}
+ {{- range .Values.alert.persistentVolumeClaim.accessModes }}
- {{ . | quote }}
{{- end }}
storageClassName: {{ .Values.alert.persistentVolumeClaim.storageClassName | quote }}
diff --git a/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-api.yaml b/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-api.yaml
index c1074cc..4829296 100644
--- a/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-api.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/pvc-dolphinscheduler-api.yaml
@@ -25,7 +25,7 @@ metadata:
app.kubernetes.io/managed-by: {{ .Release.Service }}
spec:
accessModes:
- {{- range .Values.api.persistentVolumeClaim.accessModes }}
+ {{- range .Values.api.persistentVolumeClaim.accessModes }}
- {{ . | quote }}
{{- end }}
storageClassName: {{ .Values.api.persistentVolumeClaim.storageClassName | quote }}
diff --git a/docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-frontend.yaml b/docker/kubernetes/dolphinscheduler/templates/secret-resource-s3.yaml
similarity index 69%
rename from docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-frontend.yaml
rename to docker/kubernetes/dolphinscheduler/templates/secret-resource-s3.yaml
index 60d0d6e..a0c9376 100644
--- a/docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-frontend.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/secret-resource-s3.yaml
@@ -14,22 +14,16 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
+{{- if eq .Values.resource.resourceStorageType "S3" }}
apiVersion: v1
-kind: Service
+kind: Secret
metadata:
- name: {{ include "dolphinscheduler.fullname" . }}-frontend
+ name: {{ printf "%s-%s" .Release.Name "resource-s3" }}
labels:
- app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-frontend
+ app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-resource-s3
app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
-spec:
- ports:
- - port: 8888
- targetPort: tcp-port
- protocol: TCP
- name: tcp-port
- selector:
- app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-frontend
- app.kubernetes.io/instance: {{ .Release.Name }}
- app.kubernetes.io/managed-by: {{ .Release.Service }}
- app.kubernetes.io/component: frontend
\ No newline at end of file
+type: Opaque
+data:
+ fs-s3a-secret-key: {{ .Values.resource.fsS3aSecretKey | b64enc | quote }}
+{{- end }}
\ No newline at end of file
diff --git a/docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml b/docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml
index b142fac..7f42205 100644
--- a/docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-master.yaml
@@ -50,59 +50,6 @@ spec:
{{- if .Values.master.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
- initContainers:
- - name: init-zookeeper
- image: busybox:1.31.0
- command:
- - /bin/sh
- - -ec
- - |
- echo "${ZOOKEEPER_QUORUM}" | awk -F ',' 'BEGIN{ i=1 }{ while( i <= NF ){ print $i; i++ } }' | while read line; do
- while ! nc -z ${line%:*} ${line#*:}; do
- counter=$((counter+1))
- if [ $counter == 5 ]; then
- echo "Error: Couldn't connect to zookeeper."
- exit 1
- fi
- echo "Trying to connect to zookeeper at ${line}. Attempt $counter."
- sleep 60
- done
- done
- env:
- - name: ZOOKEEPER_QUORUM
- {{- if .Values.zookeeper.enabled }}
- value: "{{ template "dolphinscheduler.zookeeper.quorum" . }}"
- {{- else }}
- value: {{ .Values.externalZookeeper.zookeeperQuorum }}
- {{- end }}
- - name: init-postgresql
- image: busybox:1.31.0
- command:
- - /bin/sh
- - -ec
- - |
- while ! nc -z ${POSTGRESQL_HOST} ${POSTGRESQL_PORT}; do
- counter=$((counter+1))
- if [ $counter == 5 ]; then
- echo "Error: Couldn't connect to postgresql."
- exit 1
- fi
- echo "Trying to connect to postgresql at ${POSTGRESQL_HOST}:${POSTGRESQL_PORT}. Attempt $counter."
- sleep 60
- done
- env:
- - name: POSTGRESQL_HOST
- {{- if .Values.postgresql.enabled }}
- value: {{ template "dolphinscheduler.postgresql.fullname" . }}
- {{- else }}
- value: {{ .Values.externalDatabase.host | quote }}
- {{- end }}
- - name: POSTGRESQL_PORT
- {{- if .Values.postgresql.enabled }}
- value: "5432"
- {{- else }}
- value: {{ .Values.externalDatabase.port }}
- {{- end }}
containers:
- name: {{ include "dolphinscheduler.fullname" . }}-master
image: {{ include "dolphinscheduler.image.repository" . | quote }}
@@ -155,30 +102,25 @@ spec:
configMapKeyRef:
name: {{ include "dolphinscheduler.fullname" . }}-master
key: MASTER_LISTEN_PORT
- - name: DOLPHINSCHEDULER_DATA_BASEDIR_PATH
- valueFrom:
- configMapKeyRef:
- name: {{ include "dolphinscheduler.fullname" . }}-master
- key: DOLPHINSCHEDULER_DATA_BASEDIR_PATH
- - name: POSTGRESQL_HOST
+ - name: DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "dolphinscheduler.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- - name: POSTGRESQL_PORT
+ - name: DATABASE_PORT
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port }}
{{- end }}
- - name: POSTGRESQL_USERNAME
+ - name: DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.username | quote }}
{{- end }}
- - name: POSTGRESQL_PASSWORD
+ - name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.postgresql.enabled }}
@@ -188,7 +130,7 @@ spec:
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
{{- end }}
- - name: POSTGRESQL_DATABASE
+ - name: DATABASE_DATABASE
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
@@ -200,6 +142,12 @@ spec:
{{- else }}
value: {{ .Values.externalZookeeper.zookeeperQuorum }}
{{- end }}
+ - name: ZOOKEEPER_ROOT
+ {{- if .Values.zookeeper.enabled }}
+ value: {{ .Values.zookeeper.zookeeperRoot }}
+ {{- else }}
+ value: {{ .Values.externalZookeeper.zookeeperRoot }}
+ {{- end }}
{{- if .Values.master.livenessProbe.enabled }}
livenessProbe:
exec:
diff --git a/docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml b/docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml
index 198cef4..a390a7b 100644
--- a/docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/statefulset-dolphinscheduler-worker.yaml
@@ -50,59 +50,6 @@ spec:
{{- if .Values.worker.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
- initContainers:
- - name: init-zookeeper
- image: busybox:1.31.0
- command:
- - /bin/sh
- - -ec
- - |
- echo "${ZOOKEEPER_QUORUM}" | awk -F ',' 'BEGIN{ i=1 }{ while( i <= NF ){ print $i; i++ } }' | while read line; do
- while ! nc -z ${line%:*} ${line#*:}; do
- counter=$((counter+1))
- if [ $counter == 5 ]; then
- echo "Error: Couldn't connect to zookeeper."
- exit 1
- fi
- echo "Trying to connect to zookeeper at ${line}. Attempt $counter."
- sleep 60
- done
- done
- env:
- - name: ZOOKEEPER_QUORUM
- {{- if .Values.zookeeper.enabled }}
- value: "{{ template "dolphinscheduler.zookeeper.quorum" . }}"
- {{- else }}
- value: {{ .Values.externalZookeeper.zookeeperQuorum }}
- {{- end }}
- - name: init-postgresql
- image: busybox:1.31.0
- command:
- - /bin/sh
- - -ec
- - |
- while ! nc -z ${POSTGRESQL_HOST} ${POSTGRESQL_PORT}; do
- counter=$((counter+1))
- if [ $counter == 5 ]; then
- echo "Error: Couldn't connect to postgresql."
- exit 1
- fi
- echo "Trying to connect to postgresql at ${POSTGRESQL_HOST}:${POSTGRESQL_PORT}. Attempt $counter."
- sleep 60
- done
- env:
- - name: POSTGRESQL_HOST
- {{- if .Values.postgresql.enabled }}
- value: {{ template "dolphinscheduler.postgresql.fullname" . }}
- {{- else }}
- value: {{ .Values.externalDatabase.host | quote }}
- {{- end }}
- - name: POSTGRESQL_PORT
- {{- if .Values.postgresql.enabled }}
- value: "5432"
- {{- else }}
- value: {{ .Values.externalDatabase.port }}
- {{- end }}
containers:
- name: {{ include "dolphinscheduler.fullname" . }}-worker
image: {{ include "dolphinscheduler.image.repository" . | quote }}
@@ -112,7 +59,7 @@ spec:
- containerPort: {{ .Values.worker.configmap.WORKER_LISTEN_PORT }}
name: "worker-port"
- containerPort: 50051
- name: "logs-port"
+ name: "logger-port"
imagePullPolicy: {{ .Values.image.pullPolicy }}
env:
- name: TZ
@@ -122,11 +69,6 @@ spec:
configMapKeyRef:
name: {{ include "dolphinscheduler.fullname" . }}-worker
key: WORKER_EXEC_THREADS
- - name: WORKER_FETCH_TASK_NUM
- valueFrom:
- configMapKeyRef:
- name: {{ include "dolphinscheduler.fullname" . }}-worker
- key: WORKER_FETCH_TASK_NUM
- name: WORKER_HEARTBEAT_INTERVAL
valueFrom:
configMapKeyRef:
@@ -155,27 +97,27 @@ spec:
- name: DOLPHINSCHEDULER_DATA_BASEDIR_PATH
valueFrom:
configMapKeyRef:
- name: {{ include "dolphinscheduler.fullname" . }}-master
+ name: {{ include "dolphinscheduler.fullname" . }}-worker
key: DOLPHINSCHEDULER_DATA_BASEDIR_PATH
- - name: POSTGRESQL_HOST
+ - name: DATABASE_HOST
{{- if .Values.postgresql.enabled }}
value: {{ template "dolphinscheduler.postgresql.fullname" . }}
{{- else }}
value: {{ .Values.externalDatabase.host | quote }}
{{- end }}
- - name: POSTGRESQL_PORT
+ - name: DATABASE_PORT
{{- if .Values.postgresql.enabled }}
value: "5432"
{{- else }}
value: {{ .Values.externalDatabase.port }}
{{- end }}
- - name: POSTGRESQL_USERNAME
+ - name: DATABASE_USERNAME
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlUsername }}
{{- else }}
value: {{ .Values.externalDatabase.username | quote }}
{{- end }}
- - name: POSTGRESQL_PASSWORD
+ - name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
{{- if .Values.postgresql.enabled }}
@@ -185,7 +127,7 @@ spec:
name: {{ printf "%s-%s" .Release.Name "externaldb" }}
key: db-password
{{- end }}
- - name: POSTGRESQL_DATABASE
+ - name: DATABASE_DATABASE
{{- if .Values.postgresql.enabled }}
value: {{ .Values.postgresql.postgresqlDatabase }}
{{- else }}
@@ -197,6 +139,44 @@ spec:
{{- else }}
value: {{ .Values.externalZookeeper.zookeeperQuorum }}
{{- end }}
+ - name: ZOOKEEPER_ROOT
+ {{- if .Values.zookeeper.enabled }}
+ value: {{ .Values.zookeeper.zookeeperRoot }}
+ {{- else }}
+ value: {{ .Values.externalZookeeper.zookeeperRoot }}
+ {{- end }}
+ - name: RESOURCE_STORAGE_TYPE
+ valueFrom:
+ configMapKeyRef:
+ key: RESOURCE_STORAGE_TYPE
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ - name: RESOURCE_UPLOAD_PATH
+ valueFrom:
+ configMapKeyRef:
+ key: RESOURCE_UPLOAD_PATH
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ - name: FS_DEFAULT_FS
+ valueFrom:
+ configMapKeyRef:
+ key: FS_DEFAULT_FS
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ {{- if eq .Values.resource.resourceStorageType "S3" }}
+ - name: FS_S3A_ENDPOINT
+ valueFrom:
+ configMapKeyRef:
+ key: FS_S3A_ENDPOINT
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ - name: FS_S3A_ACCESS_KEY
+ valueFrom:
+ configMapKeyRef:
+ key: FS_S3A_ACCESS_KEY
+ name: {{ include "dolphinscheduler.fullname" . }}-resource
+ - name: FS_S3A_SECRET_KEY
+ valueFrom:
+ secretKeyRef:
+ key: fs-s3a-secret-key
+ name: {{ printf "%s-%s" .Release.Name "resource-s3" }}
+ {{- end }}
{{- if .Values.worker.livenessProbe.enabled }}
livenessProbe:
exec:
diff --git a/docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-worker-headless.yaml b/docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-worker-headless.yaml
index fb3b85b..5c4e754 100644
--- a/docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-worker-headless.yaml
+++ b/docker/kubernetes/dolphinscheduler/templates/svc-dolphinscheduler-worker-headless.yaml
@@ -30,9 +30,9 @@ spec:
protocol: TCP
name: worker-port
- port: 50051
- targetPort: logs-port
+ targetPort: logger-port
protocol: TCP
- name: logs-port
+ name: logger-port
selector:
app.kubernetes.io/name: {{ include "dolphinscheduler.fullname" . }}-worker
app.kubernetes.io/instance: {{ .Release.Name }}
diff --git a/docker/kubernetes/dolphinscheduler/values.yaml b/docker/kubernetes/dolphinscheduler/values.yaml
index 3cb35c1..5206450 100644
--- a/docker/kubernetes/dolphinscheduler/values.yaml
+++ b/docker/kubernetes/dolphinscheduler/values.yaml
@@ -55,21 +55,28 @@ externalDatabase:
# If not exists external zookeeper, by default, Dolphinscheduler's zookeeper will use it.
zookeeper:
enabled: true
- taskQueue: "zookeeper"
- config:
- ZOO_4LW_COMMANDS_WHITELIST: srvr,ruok,wchs,cons
+ fourlwCommandsWhitelist: srvr,ruok,wchs,cons
service:
port: "2181"
persistence:
enabled: false
size: "20Gi"
storageClass: "-"
+ zookeeperRoot: /dolphinscheduler
# If exists external zookeeper, and set zookeeper.enable value to false.
# If zookeeper.enable is false, Dolphinscheduler's zookeeper will use it.
externalZookeeper:
- taskQueue: "zookeeper"
zookeeperQuorum: "127.0.0.1:2181"
+ zookeeperRoot: /dolphinscheduler
+
+resource:
+ resourceStorageType: "HDFS"
+ resourceUploadPath: "/dolphinscheduler"
+ fsDefaultFS: "file:///"
+ fsS3aEndpoint: "s3.xxx.amazonaws.com"
+ fsS3aAccessKey: "xxxxxxx"
+ fsS3aSecretKey: "xxxxxxx"
master:
podManagementPolicy: "Parallel"
@@ -85,8 +92,6 @@ master:
# If specified, the pod's scheduling constraints.
# More info: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#affinity-v1-core
affinity: {}
- ## Periodic probe of container liveness. Container will be restarted if the probe fails. Cannot be updated.
- ## More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
configmap:
MASTER_EXEC_THREADS: "100"
MASTER_EXEC_TASK_NUM: "20"
@@ -96,6 +101,8 @@ master:
MASTER_MAX_CPULOAD_AVG: "100"
MASTER_RESERVED_MEMORY: "0.1"
MASTER_LISTEN_PORT: "5678"
+ ## Periodic probe of container liveness. Container will be restarted if the probe fails. Cannot be updated.
+ ## More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
livenessProbe:
enabled: true
initialDelaySeconds: "30"
@@ -137,6 +144,25 @@ worker:
# If specified, the pod's scheduling constraints.
# More info: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#affinity-v1-core
affinity: {}
+ configmap:
+ WORKER_EXEC_THREADS: "100"
+ WORKER_HEARTBEAT_INTERVAL: "10"
+ WORKER_MAX_CPULOAD_AVG: "100"
+ WORKER_RESERVED_MEMORY: "0.1"
+ WORKER_LISTEN_PORT: "1234"
+ WORKER_GROUP: "default"
+ DOLPHINSCHEDULER_DATA_BASEDIR_PATH: "/tmp/dolphinscheduler"
+ DOLPHINSCHEDULER_ENV:
+ - "export HADOOP_HOME=/opt/soft/hadoop"
+ - "export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop"
+ - "export SPARK_HOME1=/opt/soft/spark1"
+ - "export SPARK_HOME2=/opt/soft/spark2"
+ - "export PYTHON_HOME=/usr/bin/python"
+ - "export JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk"
+ - "export HIVE_HOME=/opt/soft/hive"
+ - "export FLINK_HOME=/opt/soft/flink"
+ - "export DATAX_HOME=/opt/soft/datax/bin/datax.py"
+ - "export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$PATH"
## Periodic probe of container liveness. Container will be restarted if the probe fails. Cannot be updated.
## More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
livenessProbe:
@@ -155,25 +181,6 @@ worker:
timeoutSeconds: "5"
failureThreshold: "3"
successThreshold: "1"
- configmap:
- WORKER_EXEC_THREADS: "100"
- WORKER_HEARTBEAT_INTERVAL: "10"
- WORKER_FETCH_TASK_NUM: "3"
- WORKER_MAX_CPULOAD_AVG: "100"
- WORKER_RESERVED_MEMORY: "0.1"
- WORKER_LISTEN_PORT: "1234"
- WORKER_GROUP: "default"
- DOLPHINSCHEDULER_DATA_BASEDIR_PATH: "/tmp/dolphinscheduler"
- DOLPHINSCHEDULER_ENV:
- - "export HADOOP_HOME=/opt/soft/hadoop"
- - "export HADOOP_CONF_DIR=/opt/soft/hadoop/etc/hadoop"
- - "export SPARK_HOME1=/opt/soft/spark1"
- - "export SPARK_HOME2=/opt/soft/spark2"
- - "export PYTHON_HOME=/opt/soft/python"
- - "export JAVA_HOME=/opt/soft/java"
- - "export HIVE_HOME=/opt/soft/hive"
- - "export FLINK_HOME=/opt/soft/flink"
- - "export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$PATH"
## volumeClaimTemplates is a list of claims that pods are allowed to reference.
## The StatefulSet controller is responsible for mapping network identities to claims in a way that maintains the identity of a pod.
## Every claim in this list must have at least one matching (by name) volumeMount in one container in the template.
@@ -213,8 +220,6 @@ alert:
# If specified, the pod's scheduling constraints.
# More info: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#affinity-v1-core
affinity: {}
- ## Periodic probe of container liveness. Container will be restarted if the probe fails. Cannot be updated.
- ## More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
configmap:
XLS_FILE_PATH: "/tmp/xls"
MAIL_SERVER_HOST: ""
@@ -230,51 +235,6 @@ alert:
ENTERPRISE_WECHAT_SECRET: ""
ENTERPRISE_WECHAT_AGENT_ID: ""
ENTERPRISE_WECHAT_USERS: ""
- livenessProbe:
- enabled: true
- initialDelaySeconds: "30"
- periodSeconds: "30"
- timeoutSeconds: "5"
- failureThreshold: "3"
- successThreshold: "1"
- ## Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. Cannot be updated.
- ## More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
- readinessProbe:
- enabled: true
- initialDelaySeconds: "30"
- periodSeconds: "30"
- timeoutSeconds: "5"
- failureThreshold: "3"
- successThreshold: "1"
- ## volumeClaimTemplates is a list of claims that pods are allowed to reference.
- ## The StatefulSet controller is responsible for mapping network identities to claims in a way that maintains the identity of a pod.
- ## Every claim in this list must have at least one matching (by name) volumeMount in one container in the template.
- ## A claim in this list takes precedence over any volumes in the template, with the same name.
- persistentVolumeClaim:
- enabled: false
- accessModes:
- - "ReadWriteOnce"
- storageClassName: "-"
- storage: "20Gi"
-
-api:
- strategy:
- type: "RollingUpdate"
- rollingUpdate:
- maxSurge: "25%"
- maxUnavailable: "25%"
- replicas: "1"
- # NodeSelector is a selector which must be true for the pod to fit on a node.
- # Selector which must match a node's labels for the pod to be scheduled on that node.
- # More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
- nodeSelector: {}
- # Tolerations are appended (excluding duplicates) to pods running with this RuntimeClass during admission,
- # effectively unioning the set of nodes tolerated by the pod and the RuntimeClass.
- tolerations: []
- # Affinity is a group of affinity scheduling rules.
- # If specified, the pod's scheduling constraints.
- # More info: https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/#affinity-v1-core
- affinity: {}
## Periodic probe of container liveness. Container will be restarted if the probe fails. Cannot be updated.
## More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
livenessProbe:
@@ -304,7 +264,7 @@ api:
storageClassName: "-"
storage: "20Gi"
-frontend:
+api:
strategy:
type: "RollingUpdate"
rollingUpdate: