You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@predictionio.apache.org by ha...@apache.org on 2019/04/07 13:16:45 UTC
[predictionio] branch develop updated: [PIO-198] add helm chart
(#502)
This is an automated email from the ASF dual-hosted git repository.
hagino pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/predictionio.git
The following commit(s) were added to refs/heads/develop by this push:
new 794d5ef [PIO-198] add helm chart (#502)
794d5ef is described below
commit 794d5ef4d114e357613a75a60fc7f9f81b121481
Author: Shinsuke Sugaya <sh...@gmail.com>
AuthorDate: Sun Apr 7 22:16:38 2019 +0900
[PIO-198] add helm chart (#502)
---
docker/charts/README.md | 59 +++++++++++++
docker/charts/postgresql.yaml | 23 +++++
docker/charts/predictionio/.helmignore | 38 +++++++++
docker/charts/predictionio/Chart.yaml | 27 ++++++
docker/charts/predictionio/templates/NOTES.txt | 31 +++++++
docker/charts/predictionio/templates/_helpers.tpl | 36 ++++++++
.../predictionio/templates/pio-deployment.yaml | 75 ++++++++++++++++
.../charts/predictionio/templates/pio-service.yaml | 35 ++++++++
docker/charts/predictionio/values.yaml | 53 ++++++++++++
docker/charts/predictionio_postgresql.yaml | 41 +++++++++
docker/charts/spark/.helmignore | 38 +++++++++
docker/charts/spark/Chart.yaml | 30 +++++++
docker/charts/spark/README.md | 99 ++++++++++++++++++++++
docker/charts/spark/templates/NOTES.txt | 24 ++++++
docker/charts/spark/templates/_helpers.tpl | 43 ++++++++++
.../spark/templates/spark-master-deployment.yaml | 95 +++++++++++++++++++++
docker/charts/spark/templates/spark-sql-test.yaml | 30 +++++++
.../spark/templates/spark-worker-deployment.yaml | 57 +++++++++++++
.../charts/spark/templates/spark-worker-hpa.yaml | 39 +++++++++
docker/charts/spark/values.yaml | 63 ++++++++++++++
20 files changed, 936 insertions(+)
diff --git a/docker/charts/README.md b/docker/charts/README.md
new file mode 100644
index 0000000..2590d7a
--- /dev/null
+++ b/docker/charts/README.md
@@ -0,0 +1,59 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+Helm Charts for Apache PredictionIO
+============================
+
+## Overview
+
+Helm Charts are packages of pre-configured Kubernetes resources.
+Using charts, you can install and manage PredictionIO in the Kubernetes.
+
+## Usage
+
+### Install PredictionIO with PostgreSQL
+
+To install PostgreSQL and PredictionIO, run `helm install` command:
+
+```
+helm install --name my-postgresql stable/postgresql -f postgresql.yaml
+helm install --name my-pio ./predictionio -f predictionio_postgresql.yaml
+```
+
+`postgresql.yaml` and `predictionio_postgresql.yaml` are configuration files for charts.
+To access Jupyter for PredictionIO, run `kubectl port-forward` and then open `http://localhost:8888/`.
+
+```
+export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=predictionio,app.kubernetes.io/instance=my-pio" -o jsonpath="{.items[0].metadata.name}")
+kubectl port-forward $POD_NAME 8888:8888
+```
+
+
+### Install Spark Cluster
+
+To install Spark cluster, run the following command:
+
+```
+helm install --name my-spark ./spark
+```
+
+To train a model, run `pio train` as below:
+
+```
+pio train -- --master spark://my-spark-master:7077
+```
+
diff --git a/docker/charts/postgresql.yaml b/docker/charts/postgresql.yaml
new file mode 100644
index 0000000..6d4e383
--- /dev/null
+++ b/docker/charts/postgresql.yaml
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+postgresqlUsername: pio
+postgresqlPassword: pio
+postgresqlDatabase: pio
+
+# for testing
+persistence:
+ enabled: false
diff --git a/docker/charts/predictionio/.helmignore b/docker/charts/predictionio/.helmignore
new file mode 100644
index 0000000..4a332ba
--- /dev/null
+++ b/docker/charts/predictionio/.helmignore
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
diff --git a/docker/charts/predictionio/Chart.yaml b/docker/charts/predictionio/Chart.yaml
new file mode 100644
index 0000000..7f97bc8
--- /dev/null
+++ b/docker/charts/predictionio/Chart.yaml
@@ -0,0 +1,27 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+name: predictionio
+version: 0.1.0
+appVersion: 0.13.0
+description: Machine learning server
+home: http://predictionio.apache.org
+icon: http://predictionio.apache.org/images/logos/logo-ee2b9bb3.png
+sources:
+ - https://github.com/apache/predictionio
+maintainers:
+ - name: Shinsuke Sugaya
+ email: shinsuke@apache.org
diff --git a/docker/charts/predictionio/templates/NOTES.txt b/docker/charts/predictionio/templates/NOTES.txt
new file mode 100644
index 0000000..9a2e414
--- /dev/null
+++ b/docker/charts/predictionio/templates/NOTES.txt
@@ -0,0 +1,31 @@
+{{/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */}}
+1. Get the application URL by running these commands:
+{{- if contains "NodePort" .Values.pio.service.type }}
+ export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "predictionio.fullname" . }})
+ export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
+ echo http://$NODE_IP:$NODE_PORT
+{{- else if contains "LoadBalancer" .Values.pio.service.type }}
+ NOTE: It may take a few minutes for the LoadBalancer IP to be available.
+ You can watch the status of by running 'kubectl get svc -w {{ include "predictionio.fullname" . }}'
+ export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "predictionio.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+ echo http://$SERVICE_IP:{{ .Values.pio.service.port }}
+{{- else if contains "ClusterIP" .Values.pio.service.type }}
+ export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "predictionio.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
+ echo "Visit http://127.0.0.1:8888 to use your application"
+ kubectl port-forward $POD_NAME 8888:8888
+{{- end }}
diff --git a/docker/charts/predictionio/templates/_helpers.tpl b/docker/charts/predictionio/templates/_helpers.tpl
new file mode 100644
index 0000000..57f345d
--- /dev/null
+++ b/docker/charts/predictionio/templates/_helpers.tpl
@@ -0,0 +1,36 @@
+{{/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */}}
+{{- define "predictionio.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{- define "predictionio.fullname" -}}
+{{- if .Values.fullnameOverride -}}
+{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
+{{- $name := default .Chart.Name .Values.nameOverride -}}
+{{- if contains $name .Release.Name -}}
+{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
+{{- else -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+{{- end -}}
+{{- end -}}
+
+{{- define "predictionio.chart" -}}
+{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
diff --git a/docker/charts/predictionio/templates/pio-deployment.yaml b/docker/charts/predictionio/templates/pio-deployment.yaml
new file mode 100644
index 0000000..94ca1b6
--- /dev/null
+++ b/docker/charts/predictionio/templates/pio-deployment.yaml
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+apiVersion: apps/v1beta2
+kind: Deployment
+metadata:
+ name: {{ include "predictionio.fullname" . }}
+ labels:
+ app.kubernetes.io/name: {{ include "predictionio.name" . }}
+ helm.sh/chart: {{ include "predictionio.chart" . }}
+ app.kubernetes.io/instance: {{ .Release.Name }}
+ app.kubernetes.io/managed-by: {{ .Release.Service }}
+spec:
+ replicas: {{ .Values.pio.replicas }}
+ selector:
+ matchLabels:
+ app.kubernetes.io/name: {{ include "predictionio.name" . }}
+ app.kubernetes.io/instance: {{ .Release.Name }}
+ template:
+ metadata:
+ labels:
+ app.kubernetes.io/name: {{ include "predictionio.name" . }}
+ app.kubernetes.io/instance: {{ .Release.Name }}
+ spec:
+ containers:
+ - name: {{ .Chart.Name }}
+ image: "{{ .Values.pio.image.repository }}:{{ .Values.pio.image.tag }}"
+ imagePullPolicy: {{ .Values.pio.image.pullPolicy }}
+ env:
+{{ toYaml .Values.pio.env | indent 12 }}
+ ports:
+ - name: event
+ containerPort: 7070
+ protocol: TCP
+ - name: predict
+ containerPort: 8000
+ protocol: TCP
+ - name: jupyter
+ containerPort: 8888
+ protocol: TCP
+ livenessProbe:
+ httpGet:
+ path: /
+ port: 7070
+ readinessProbe:
+ httpGet:
+ path: /
+ port: 7070
+ resources:
+{{ toYaml .Values.pio.resources | indent 12 }}
+ {{- with .Values.pio.nodeSelector }}
+ nodeSelector:
+{{ toYaml . | indent 8 }}
+ {{- end }}
+ {{- with .Values.pio.affinity }}
+ affinity:
+{{ toYaml . | indent 8 }}
+ {{- end }}
+ {{- with .Values.pio.tolerations }}
+ tolerations:
+{{ toYaml . | indent 8 }}
+ {{- end }}
diff --git a/docker/charts/predictionio/templates/pio-service.yaml b/docker/charts/predictionio/templates/pio-service.yaml
new file mode 100644
index 0000000..d4f8130
--- /dev/null
+++ b/docker/charts/predictionio/templates/pio-service.yaml
@@ -0,0 +1,35 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ include "predictionio.fullname" . }}
+ labels:
+ app.kubernetes.io/name: {{ include "predictionio.name" . }}
+ helm.sh/chart: {{ include "predictionio.chart" . }}
+ app.kubernetes.io/instance: {{ .Release.Name }}
+ app.kubernetes.io/managed-by: {{ .Release.Service }}
+spec:
+ type: {{ .Values.pio.service.type }}
+ ports:
+ - port: {{ .Values.pio.service.port }}
+ targetPort: 8888
+ protocol: TCP
+ name: jupyter
+ selector:
+ app.kubernetes.io/name: {{ include "predictionio.name" . }}
+ app.kubernetes.io/instance: {{ .Release.Name }}
diff --git a/docker/charts/predictionio/values.yaml b/docker/charts/predictionio/values.yaml
new file mode 100644
index 0000000..db1d2d9
--- /dev/null
+++ b/docker/charts/predictionio/values.yaml
@@ -0,0 +1,53 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+pio:
+ replicas: 1
+ image:
+ repository: predictionio/pio-jupyter
+ tag: latest
+ pullPolicy: IfNotPresent
+ service:
+ type: ClusterIP
+ port: 8888
+ env:
+ - name: PIO_STORAGE_SOURCES_PGSQL_TYPE
+ value: jdbc
+ - name: PIO_STORAGE_SOURCES_PGSQL_URL
+ value: "jdbc:postgresql://postgresql/pio"
+ - name: PIO_STORAGE_SOURCES_PGSQL_USERNAME
+ value: pio
+ - name: PIO_STORAGE_SOURCES_PGSQL_PASSWORD
+ value: pio
+ - name: PIO_STORAGE_REPOSITORIES_MODELDATA_NAME
+ value: pio_model
+ - name: PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE
+ value: PGSQL
+ - name: PIO_STORAGE_REPOSITORIES_METADATA_NAME
+ value: pio_meta
+ - name: PIO_STORAGE_REPOSITORIES_METADATA_SOURCE
+ value: PGSQL
+ - name: PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME
+ value: pio_event
+ - name: PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE
+ value: PGSQL
+ - name: PYSPARK_DRIVER_PYTHON_OPTS
+ value: "notebook --NotebookApp.token=''"
+ resources: {}
+ nodeSelector: {}
+ tolerations: []
+ affinity: {}
+
diff --git a/docker/charts/predictionio_postgresql.yaml b/docker/charts/predictionio_postgresql.yaml
new file mode 100644
index 0000000..8214d5b
--- /dev/null
+++ b/docker/charts/predictionio_postgresql.yaml
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+pio:
+ env:
+ - name: PIO_STORAGE_SOURCES_PGSQL_TYPE
+ value: jdbc
+ - name: PIO_STORAGE_SOURCES_PGSQL_URL
+ value: "jdbc:postgresql://my-postgresql-postgresql:5432/pio"
+ - name: PIO_STORAGE_SOURCES_PGSQL_USERNAME
+ value: pio
+ - name: PIO_STORAGE_SOURCES_PGSQL_PASSWORD
+ value: pio
+ - name: PIO_STORAGE_REPOSITORIES_MODELDATA_NAME
+ value: pio_model
+ - name: PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE
+ value: PGSQL
+ - name: PIO_STORAGE_REPOSITORIES_METADATA_NAME
+ value: pio_meta
+ - name: PIO_STORAGE_REPOSITORIES_METADATA_SOURCE
+ value: PGSQL
+ - name: PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME
+ value: pio_event
+ - name: PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE
+ value: PGSQL
+ - name: PYSPARK_DRIVER_PYTHON_OPTS
+ value: "notebook --NotebookApp.token=''"
+
diff --git a/docker/charts/spark/.helmignore b/docker/charts/spark/.helmignore
new file mode 100644
index 0000000..4a332ba
--- /dev/null
+++ b/docker/charts/spark/.helmignore
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
diff --git a/docker/charts/spark/Chart.yaml b/docker/charts/spark/Chart.yaml
new file mode 100755
index 0000000..aee7901
--- /dev/null
+++ b/docker/charts/spark/Chart.yaml
@@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+name: spark
+version: 0.3.0
+appVersion: 2.3.2
+description: Fast and general-purpose cluster computing system.
+home: http://spark.apache.org
+icon: http://spark.apache.org/images/spark-logo-trademark.png
+sources:
+ - https://github.com/kubernetes/kubernetes/tree/master/examples/spark
+ - https://github.com/apache/spark
+maintainers:
+ - name: lachie83
+ email: lachlan.evenson@gmail.com
+ - name: Shinsuke Sugaya
+ email: shinsuke@apache.org
diff --git a/docker/charts/spark/README.md b/docker/charts/spark/README.md
new file mode 100644
index 0000000..4914f13
--- /dev/null
+++ b/docker/charts/spark/README.md
@@ -0,0 +1,99 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+# Apache Spark Helm Chart
+
+Apache Spark is a fast and general-purpose cluster computing system.
+
+* http://spark.apache.org/
+
+This chart is based on stable/spark in [Helm Charts](https://github.com/helm/charts).
+
+## Chart Details
+This chart will do the following:
+
+* 1 x Spark Master with port 8080 exposed on an external LoadBalancer
+* 3 x Spark Workers with HorizontalPodAutoscaler to scale to max 10 pods when CPU hits 50% of 100m
+* All using Kubernetes Deployments
+
+## Prerequisites
+
+* Assumes that serviceAccount tokens are available under hostname metadata. (Works on GKE by default) URL -- http://metadata/computeMetadata/v1/instance/service-accounts/default/token
+
+## Installing the Chart
+
+To install the chart with the release name `my-release`:
+
+```bash
+$ helm install --name my-release stable/spark
+```
+
+## Configuration
+
+The following table lists the configurable parameters of the Spark chart and their default values.
+
+### Spark Master
+
+| Parameter | Description | Default |
+| ----------------------- | ---------------------------------- | ---------------------------------------------------------- |
+| `Master.Name` | Spark master name | `spark-master` |
+| `Master.Image` | Container image name | `bde2020/spark-master` |
+| `Master.ImageTag` | Container image tag | `2.2.2-hadoop2.7` |
+| `Master.Replicas` | k8s deployment replicas | `1` |
+| `Master.Component` | k8s selector key | `spark-master` |
+| `Master.Cpu` | container requested cpu | `100m` |
+| `Master.Memory` | container requested memory | `512Mi` |
+| `Master.ServicePort` | k8s service port | `7077` |
+| `Master.ContainerPort` | Container listening port | `7077` |
+| `Master.DaemonMemory` | Master JVM Xms and Xmx option | `1g` |
+| `Master.ServiceType ` | Kubernetes Service type | `LoadBalancer` |
+
+### Spark WebUi
+
+| Parameter | Description | Default |
+|-----------------------|----------------------------------|----------------------------------------------------------|
+| `WebUi.Name` | Spark webui name | `spark-webui` |
+| `WebUi.ServicePort` | k8s service port | `8080` |
+| `WebUi.ContainerPort` | Container listening port | `8080` |
+
+### Spark Worker
+
+| Parameter | Description | Default |
+| ----------------------- | ------------------------------------ | ---------------------------------------------------------- |
+| `Worker.Name` | Spark worker name | `spark-worker` |
+| `Worker.Image` | Container image name | `bde2020/spark-worker` |
+| `Worker.ImageTag` | Container image tag | `2.2.2-hadoop2.7` |
+| `Worker.Replicas` | k8s hpa and deployment replicas | `3` |
+| `Worker.ReplicasMax` | k8s hpa max replicas | `10` |
+| `Worker.Component` | k8s selector key | `spark-worker` |
+| `Worker.Cpu` | container requested cpu | `100m` |
+| `Worker.Memory` | container requested memory | `512Mi` |
+| `Worker.ContainerPort` | Container listening port | `7077` |
+| `Worker.CpuTargetPercentage` | k8s hpa cpu targetPercentage | `50` |
+| `Worker.DaemonMemory` | Worker JVM Xms and Xmx setting | `1g` |
+| `Worker.ExecutorMemory` | Worker memory available for executor | `1g` |
+| `Worker.Autoscaling` | Enable horizontal pod autoscaling | `false` |
+
+
+Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`.
+
+Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,
+
+```bash
+$ helm install --name my-release -f values.yaml stable/spark
+```
+
+> **Tip**: You can use the default [values.yaml](values.yaml)
diff --git a/docker/charts/spark/templates/NOTES.txt b/docker/charts/spark/templates/NOTES.txt
new file mode 100644
index 0000000..3611bf5
--- /dev/null
+++ b/docker/charts/spark/templates/NOTES.txt
@@ -0,0 +1,24 @@
+{{/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */}}
+1. Get the Spark URL to visit by running these commands in the same shell:
+
+ NOTE: It may take a few minutes for the LoadBalancer IP to be available.
+ You can watch the status of by running 'kubectl get svc --namespace {{ .Release.Namespace }} -w {{ template "webui-fullname" . }}'
+
+ export SPARK_SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ template "webui-fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+ echo http://$SPARK_SERVICE_IP:{{ .Values.WebUi.ServicePort }}
+
diff --git a/docker/charts/spark/templates/_helpers.tpl b/docker/charts/spark/templates/_helpers.tpl
new file mode 100644
index 0000000..066c441
--- /dev/null
+++ b/docker/charts/spark/templates/_helpers.tpl
@@ -0,0 +1,43 @@
+{{/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */}}
+{{/* vim: set filetype=mustache: */}}
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Create fully qualified names.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+*/}}
+{{- define "master-fullname" -}}
+{{- $name := default .Chart.Name .Values.Master.Name -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{- define "webui-fullname" -}}
+{{- $name := default .Chart.Name .Values.WebUi.Name -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{- define "worker-fullname" -}}
+{{- $name := default .Chart.Name .Values.Worker.Name -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
diff --git a/docker/charts/spark/templates/spark-master-deployment.yaml b/docker/charts/spark/templates/spark-master-deployment.yaml
new file mode 100644
index 0000000..5c05ebf
--- /dev/null
+++ b/docker/charts/spark/templates/spark-master-deployment.yaml
@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ template "master-fullname" . }}
+ labels:
+ heritage: {{ .Release.Service | quote }}
+ release: {{ .Release.Name | quote }}
+ chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+ component: "{{ .Release.Name }}-{{ .Values.Master.Component }}"
+spec:
+ ports:
+ - port: {{ .Values.Master.ServicePort }}
+ targetPort: {{ .Values.Master.ContainerPort }}
+ selector:
+ component: "{{ .Release.Name }}-{{ .Values.Master.Component }}"
+ type: {{ .Values.Master.ServiceType }}
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: {{ template "webui-fullname" . }}
+ labels:
+ heritage: {{ .Release.Service | quote }}
+ release: {{ .Release.Name | quote }}
+ chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+ component: "{{ .Release.Name }}-{{ .Values.Master.Component }}"
+spec:
+ ports:
+ - port: {{ .Values.WebUi.ServicePort }}
+ targetPort: {{ .Values.WebUi.ContainerPort }}
+ selector:
+ component: "{{ .Release.Name }}-{{ .Values.Master.Component }}"
+ type: {{ .Values.WebUi.ServiceType }}
+---
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+ name: {{ template "master-fullname" . }}
+ labels:
+ heritage: {{ .Release.Service | quote }}
+ release: {{ .Release.Name | quote }}
+ chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+ component: "{{ .Release.Name }}-{{ .Values.Master.Component }}"
+spec:
+ replicas: {{ default 1 .Values.Master.Replicas }}
+ strategy:
+ type: RollingUpdate
+ selector:
+ matchLabels:
+ component: "{{ .Release.Name }}-{{ .Values.Master.Component }}"
+ template:
+ metadata:
+ labels:
+ heritage: {{ .Release.Service | quote }}
+ release: {{ .Release.Name | quote }}
+ chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+ component: "{{ .Release.Name }}-{{ .Values.Master.Component }}"
+ spec:
+ containers:
+ - name: {{ template "master-fullname" . }}
+ image: "{{ .Values.Master.Image }}:{{ .Values.Master.ImageTag }}"
+ command: ["/bin/sh","-c"]
+ args: ["echo $(hostname -i) {{ template "master-fullname" . }} >> /etc/hosts; {{ .Values.Spark.Path }}/bin/spark-class org.apache.spark.deploy.master.Master"]
+ ports:
+ - containerPort: {{ .Values.Master.ContainerPort }}
+ - containerPort: {{ .Values.WebUi.ContainerPort }}
+ resources:
+ requests:
+ cpu: "{{ .Values.Master.Cpu }}"
+ memory: "{{ .Values.Master.Memory }}"
+ env:
+ - name: SPARK_DAEMON_MEMORY
+ value: {{ default "1g" .Values.Master.DaemonMemory | quote }}
+ - name: SPARK_MASTER_HOST
+ value: {{ template "master-fullname" . }}
+ - name: SPARK_MASTER_PORT
+ value: {{ .Values.Master.ServicePort | quote }}
+ - name: SPARK_MASTER_WEBUI_PORT
+ value: {{ .Values.WebUi.ContainerPort | quote }}
diff --git a/docker/charts/spark/templates/spark-sql-test.yaml b/docker/charts/spark/templates/spark-sql-test.yaml
new file mode 100644
index 0000000..d1a5cae
--- /dev/null
+++ b/docker/charts/spark/templates/spark-sql-test.yaml
@@ -0,0 +1,30 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+apiVersion: v1
+kind: Pod
+metadata:
+ name: "{{ .Release.Name }}-sql-test-{{ randAlphaNum 5 | lower }}"
+ annotations:
+ "helm.sh/hook": test-success
+spec:
+ containers:
+ - name: {{ .Release.Name }}-sql-test
+ image: {{ .Values.Master.Image }}:{{ .Values.Master.ImageTag }}
+ command: ["{{ .Values.Spark.Path }}/bin/spark-sql", "--master",
+ "spark://{{ .Release.Name }}-master:{{ .Values.Master.ServicePort }}", "-e",
+ "show databases;"]
+ restartPolicy: Never
diff --git a/docker/charts/spark/templates/spark-worker-deployment.yaml b/docker/charts/spark/templates/spark-worker-deployment.yaml
new file mode 100644
index 0000000..a44cc53
--- /dev/null
+++ b/docker/charts/spark/templates/spark-worker-deployment.yaml
@@ -0,0 +1,57 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+ name: {{ template "worker-fullname" . }}
+ labels:
+ heritage: {{ .Release.Service | quote }}
+ release: {{ .Release.Name | quote }}
+ chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+ component: "{{ .Release.Name }}-{{ .Values.Worker.Component }}"
+spec:
+ replicas: {{ default 1 .Values.Worker.Replicas }}
+ strategy:
+ type: RollingUpdate
+ selector:
+ matchLabels:
+ component: "{{ .Release.Name }}-{{ .Values.Worker.Component }}"
+ template:
+ metadata:
+ labels:
+ heritage: {{ .Release.Service | quote }}
+ release: {{ .Release.Name | quote }}
+ chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+ component: "{{ .Release.Name }}-{{ .Values.Worker.Component }}"
+ spec:
+ containers:
+ - name: {{ template "worker-fullname" . }}
+ image: "{{ .Values.Worker.Image }}:{{ .Values.Worker.ImageTag }}"
+ command: ["{{ .Values.Spark.Path }}/bin/spark-class", "org.apache.spark.deploy.worker.Worker", "spark://{{ template "master-fullname" . }}:{{ .Values.Master.ServicePort }}"]
+ ports:
+ - containerPort: {{ .Values.Worker.ContainerPort }}
+ resources:
+ requests:
+ cpu: "{{ .Values.Worker.Cpu }}"
+ memory: "{{ .Values.Worker.Memory }}"
+ env:
+ - name: SPARK_DAEMON_MEMORY
+ value: {{ default "1g" .Values.Worker.DaemonMemory | quote }}
+ - name: SPARK_WORKER_MEMORY
+ value: {{ default "1g" .Values.Worker.ExecutorMemory | quote }}
+ - name: SPARK_WORKER_WEBUI_PORT
+ value: {{ .Values.WebUi.ContainerPort | quote }}
diff --git a/docker/charts/spark/templates/spark-worker-hpa.yaml b/docker/charts/spark/templates/spark-worker-hpa.yaml
new file mode 100644
index 0000000..68c7d5d
--- /dev/null
+++ b/docker/charts/spark/templates/spark-worker-hpa.yaml
@@ -0,0 +1,39 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+{{- if .Values.Worker.Autoscaling.Enabled }}
+apiVersion: autoscaling/v2beta1
+kind: HorizontalPodAutoscaler
+metadata:
+ labels:
+ heritage: {{ .Release.Service | quote }}
+ release: {{ .Release.Name | quote }}
+ chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
+ component: "{{ .Release.Name }}-{{ .Values.Worker.Component }}"
+ name: {{ template "worker-fullname" . }}
+spec:
+ scaleTargetRef:
+ apiVersion: apps/v1beta1
+ kind: Deployment
+ name: {{ template "worker-fullname" . }}
+ minReplicas: {{ .Values.Worker.Replicas }}
+ maxReplicas: {{ .Values.Worker.ReplicasMax }}
+ metrics:
+ - type: Resource
+ resource:
+ name: cpu
+ targetAverageUtilization: {{ .Values.Worker.CpuTargetPercentage }}
+{{- end }}
diff --git a/docker/charts/spark/values.yaml b/docker/charts/spark/values.yaml
new file mode 100644
index 0000000..c366aa1
--- /dev/null
+++ b/docker/charts/spark/values.yaml
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Default values for spark.
+# This is a YAML-formatted file.
+# Declare name/value pairs to be passed into your templates.
+# name: value
+
+Spark:
+ Path: "/spark"
+
+Master:
+ Name: master
+ Image: "bde2020/spark-master"
+ ImageTag: "2.2.2-hadoop2.7"
+ Replicas: 1
+ Component: "spark-master"
+ Cpu: "100m"
+ Memory: "512Mi"
+ ServicePort: 7077
+ ContainerPort: 7077
+ # Set Master JVM memory. Default 1g
+ # DaemonMemory: 1g
+ ServiceType: LoadBalancer
+
+WebUi:
+ Name: webui
+ ServicePort: 8080
+ ContainerPort: 8080
+ ServiceType: LoadBalancer
+
+Worker:
+ Name: worker
+ Image: "bde2020/spark-worker"
+ ImageTag: "2.2.2-hadoop2.7"
+ Replicas: 3
+ Component: "spark-worker"
+ Cpu: "100m"
+ Memory: "512Mi"
+ ContainerPort: 8081
+ # Set Worker JVM memory. Default 1g
+ # DaemonMemory: 1g
+ # Set how much total memory workers have to give executors
+ # ExecutorMemory: 1g
+ Autoscaling:
+ Enabled: false
+ ReplicasMax: 10
+ CpuTargetPercentage: 50
+