You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@streampipes.apache.org by wi...@apache.org on 2020/12/10 08:36:25 UTC
[incubator-streampipes-installer] branch dev updated (f979bcf ->
3dac44e)
This is an automated email from the ASF dual-hosted git repository.
wiener pushed a change to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes-installer.git.
from f979bcf [STREAMPIPES-263] add extensions-all-jvm service
new cf7f21b [STREAMPIPES-264] use host path volume for persistent storage for local testing
new 3dac44e Add timescaledb test instance to cli
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../docker-compose.dev.yml | 2 +-
.../{postgis => timescaledb}/docker-compose.yml | 13 +++--
k8s/README.md | 67 ++++++++++++++++------
.../core/{backend => }/backend-deployment.yaml | 28 +++++----
.../backend-pvc.yaml => backend-pv-pvc.yaml} | 21 ++++++-
.../core/{backend => }/backend-service.yaml | 2 +-
k8s/templates/core/{ui => }/ui-ingress.yaml | 2 +-
k8s/templates/core/{ui => }/ui-service.yaml | 2 +-
.../core/{ui/ui-deployment.yaml => ui.yaml} | 13 -----
.../connect-worker-deployment.yaml | 22 +------
.../extensions/pipeline-elements/files-pvc.yaml | 22 ++++++-
.../pipeline-elements-all-flink-deployment.yaml | 7 ---
.../pipeline-elements-all-jvm-deployment.yaml | 5 --
.../pipeline-elements-all-jvm-service.yaml | 2 +-
.../sources-vehicle-simulator-deployment.yaml | 3 -
.../sources-watertank-simulator-deployment.yaml | 5 +-
.../sources-watertank-simulator-service.yaml | 29 ----------
.../external/activemq/activemq-deployment.yaml | 26 +++++++++
.../external/activemq/activemq-service.yaml | 2 +-
.../external/consul/consul-deployment.yaml | 20 +++++++
.../consul/{consul-pvc.yaml => consul-pv-pvc.yaml} | 21 ++++++-
.../{couchdb-pvc.yaml => couchdb-pv-pvc.yaml} | 21 ++++++-
.../external/flink/jobmanager-service.yaml | 2 +-
.../external/flink/taskmanager-service.yaml | 29 ----------
k8s/templates/external/influxdb/influxdb-pvc.yaml | 21 ++++++-
k8s/templates/external/kafka/kafka-deployment.yaml | 2 +
k8s/templates/external/kafka/kafka-pvc.yaml | 23 +++++++-
k8s/templates/external/kafka/kafka-service.yaml | 2 +
.../external/zookeeper/zookeeper-deployment.yaml | 2 +
.../external/zookeeper/zookeeper-pvc.yaml | 23 +++++++-
.../external/zookeeper/zookeeper-service.yaml | 4 +-
k8s/values.yaml | 11 ++--
32 files changed, 281 insertions(+), 173 deletions(-)
copy cli/deploy/standalone/{postgres => timescaledb}/docker-compose.dev.yml (98%)
copy cli/deploy/standalone/{postgis => timescaledb}/docker-compose.yml (84%)
rename k8s/templates/core/{backend => }/backend-deployment.yaml (72%)
rename k8s/templates/core/{backend/backend-pvc.yaml => backend-pv-pvc.yaml} (65%)
rename k8s/templates/core/{backend => }/backend-service.yaml (97%)
rename k8s/templates/core/{ui => }/ui-ingress.yaml (97%)
rename k8s/templates/core/{ui => }/ui-service.yaml (97%)
rename k8s/templates/core/{ui/ui-deployment.yaml => ui.yaml} (68%)
delete mode 100644 k8s/templates/extensions/pipeline-elements/sources-watertank-simulator-service.yaml
rename k8s/templates/external/consul/{consul-pvc.yaml => consul-pv-pvc.yaml} (65%)
rename k8s/templates/external/couchdb/{couchdb-pvc.yaml => couchdb-pv-pvc.yaml} (65%)
delete mode 100644 k8s/templates/external/flink/taskmanager-service.yaml
[incubator-streampipes-installer] 01/02: [STREAMPIPES-264] use host
path volume for persistent storage for local testing
Posted by wi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wiener pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes-installer.git
commit cf7f21b7f51b90e43ad4d99b751a046cd92e7726
Author: Patrick Wiener <wi...@fzi.de>
AuthorDate: Thu Dec 10 09:35:00 2020 +0100
[STREAMPIPES-264] use host path volume for persistent storage for local testing
---
k8s/README.md | 67 ++++++++++++++++------
.../core/{backend => }/backend-deployment.yaml | 28 +++++----
.../backend-pvc.yaml => backend-pv-pvc.yaml} | 21 ++++++-
.../core/{backend => }/backend-service.yaml | 2 +-
k8s/templates/core/{ui => }/ui-ingress.yaml | 2 +-
k8s/templates/core/{ui => }/ui-service.yaml | 2 +-
.../core/{ui/ui-deployment.yaml => ui.yaml} | 13 -----
.../connect-worker-deployment.yaml | 22 +------
.../extensions/pipeline-elements/files-pvc.yaml | 22 ++++++-
.../pipeline-elements-all-flink-deployment.yaml | 7 ---
.../pipeline-elements-all-jvm-deployment.yaml | 5 --
.../pipeline-elements-all-jvm-service.yaml | 2 +-
.../sources-vehicle-simulator-deployment.yaml | 3 -
.../sources-watertank-simulator-deployment.yaml | 5 +-
.../sources-watertank-simulator-service.yaml | 29 ----------
.../external/activemq/activemq-deployment.yaml | 26 +++++++++
.../external/activemq/activemq-service.yaml | 2 +-
.../external/consul/consul-deployment.yaml | 20 +++++++
.../consul/{consul-pvc.yaml => consul-pv-pvc.yaml} | 21 ++++++-
.../{couchdb-pvc.yaml => couchdb-pv-pvc.yaml} | 21 ++++++-
.../external/flink/jobmanager-service.yaml | 2 +-
.../external/flink/taskmanager-service.yaml | 29 ----------
k8s/templates/external/influxdb/influxdb-pvc.yaml | 21 ++++++-
k8s/templates/external/kafka/kafka-deployment.yaml | 2 +
k8s/templates/external/kafka/kafka-pvc.yaml | 23 +++++++-
k8s/templates/external/kafka/kafka-service.yaml | 2 +
.../external/zookeeper/zookeeper-deployment.yaml | 2 +
.../external/zookeeper/zookeeper-pvc.yaml | 23 +++++++-
.../external/zookeeper/zookeeper-service.yaml | 4 +-
k8s/values.yaml | 11 ++--
30 files changed, 273 insertions(+), 166 deletions(-)
diff --git a/k8s/README.md b/k8s/README.md
index 0c3d019..467fd9e 100644
--- a/k8s/README.md
+++ b/k8s/README.md
@@ -22,24 +22,38 @@ StreamPipes k8s is a helm chart to deploy StreamPipes on Kubernetes.
**Current version:** 0.68.0-SNAPSHOT
<!-- END do not edit -->
+We provide two helm chart templates to get you going:
+
+- **default**: a lightweight template with few pipeline elements, needs less memory
+- **full**: contains more pipeline elements, requires **>16 GB RAM** (recommended)
+
## Prerequisite
Requires Helm (https://helm.sh/) and an active connection to a kubernetes cluster with a running tiller server.
Tested with:
-* K3s v1.18.8+k3s1 (6b595318) with K8s v1.18.8
+* K8s v1.19.3
* Helm v3.1.2
+* Minikube v1.15.1 (recommended for local testing)
-## Usage
-We provide two helm chart options to get you going:
+> **NOTE**: We experienced some problems with using host path volumes in Docker Desktop environments for persistent storage. Therefore, we suggest to use minikube for local testing.
-- **default**: a light-weight option with few pipeline elements, needs less memory
-- **full**: contains more pipeline elements, requires **>16 GB RAM** (recommended)
+## Local testing
+
+We recommend using [minikube](https://minikube.sigs.k8s.io/docs/) for local testing. Follow instructions in their docs to setup test environment
-**Starting** the **default** helm chart option is as easy as simply running the following command from the root of this folder:
+Once installed, start local minikube node with a mapped host volume:
+```bash
+minikube start --mount-string ${HOME}/streampipes-k8s:/streampipes-k8s --mount --memory=4g --cpus=4
+```
+
+**Start** the a helm chart template by running the following command from the root of this folder:
> **NOTE**: Starting might take a while since we also initially pull all Docker images from Dockerhub.
```bash
helm install streampipes ./
+
+# full template only recommend if you have sufficient resources
+# helm install streampipes ./ --set deployment=full
```
After a while, all containers should successfully started, indicated by the `Running` status.
```bash
@@ -47,36 +61,53 @@ kubectl get pods
NAME READY STATUS RESTARTS AGE
activemq-66d58f47cf-2r2nb 1/1 Running 0 3m27s
backend-76ddc486c8-nswpc 1/1 Running 0 3m27s
-connect-master-7b477f9b79-8dfvr 1/1 Running 0 3m26s
connect-worker-78d89c989c-9v8zs 1/1 Running 0 3m27s
consul-55965f966b-gwb7l 1/1 Running 0 3m27s
couchdb-77db98cf7b-xnnvb 1/1 Running 0 3m27s
influxdb-b95b6479-r8wh8 1/1 Running 0 3m27s
-kafka-657b5fb77-dp2d6 1/1 Running 0 3m27s
pipeline-elements-all-jvm-79c445dbd9-m8xcs 1/1 Running 0 3m27s
-sources-watertank-simulator-6c6b8844f6-6b4d7 1/1 Running 0 3m27s
ui-b94bd9766-rm6zb 2/2 Running 0 3m27s
-zookeeper-5d9947686f-6nzgs 1/1 Running 0 3m26s
```
-After all containers are successfully started just got to your browser and visit any of the k8s cluster nodes on
-`http://<NODE_IP>` to finish the installation.
-
-> **NOTE**: If you're running Docker for Mac or Docker for Windows with a local k8s cluster, the above step to use your host IP might not work. Luckily, you can port-forward a service port to your localhost using the following command to be able to access the UI either via `http://localhost` or `http://<HOST_IP>` (you require sudo to run this command in order to bind to a privileged port).
+For **minikube users**:
+> **NOTE**: If you're running Docker Desktop or Minikube with a local k8s cluster, the above step to use your host IP might not work. Luckily, you can port-forward a service port to your localhost using the following command to be able to access the UI either via `http://localhost` or `http://<HOST_IP>` (you require sudo to run this command in order to bind to a privileged port).
```bash
kubectl port-forward svc/ui --address=0.0.0.0 80:80
```
-Starting the **full** helm chart option is almost the same:
+**Deleting** the current helm chart deployment:
```bash
-helm install streampipes ./ --set deployment=full
+helm del streampipes
```
-**Deleting** the current helm chart deployment:
+We retain the created persistent volume. You need to manually delete it:
```bash
-helm del streampipes
+rm -rf ${HOME}/streampipes-k8s
+```
+
+## Cluster Deployment
+
+We recommend to adapt the templates according to your k8s cluster setup. Especially in terms of your storage drivers for managing persistent volumes. You'll find configuration about persistent volumes and persistent volume claims in the subfolders for the various components, e.g. our `backend`:
+
+```yaml
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: backend-pv
+spec:
+ storageClassName: local-storage-backend
+ capacity:
+ storage: 50Mi
+ accessModes:
+ - {{ .Values.persistentVolumeAccessModes }}
+ persistentVolumeReclaimPolicy: {{ .Values.persistentVolumeReclaimPolicy }}
+ hostPath:
+ path: {{ .Values.hostPath }}/backend
```
+For **cluster users**:
+Just open your browser and visit any of the k8s cluster nodes on `http://<NODE_IP>` to finish the installation.
+
## Bugs and Feature Requests
If you've found a bug or have a feature that you'd love to see in StreamPipes, feel free to create an issue in our Jira:
diff --git a/k8s/templates/core/backend/backend-deployment.yaml b/k8s/templates/core/backend-deployment.yaml
similarity index 72%
rename from k8s/templates/core/backend/backend-deployment.yaml
rename to k8s/templates/core/backend-deployment.yaml
index e02b93e..a2a1b63 100644
--- a/k8s/templates/core/backend/backend-deployment.yaml
+++ b/k8s/templates/core/backend-deployment.yaml
@@ -27,28 +27,32 @@ spec:
labels:
app: backend
spec:
- initContainers:
- - name: init-wait-consul
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 consul 8600 && exit 0 || sleep 3; done; exit 1"]
- - name: init-wait-couchdb
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 couchdb 5984 && exit 0 || sleep 3; done; exit 1"]
+ hostAliases:
+ - ip: "127.0.0.1"
+ hostnames:
+ - "backend"
volumes:
- name: backend-pv
persistentVolumeClaim:
claimName: backend-pvc
- - name: files-pv
- persistentVolumeClaim:
- claimName: files-pvc
+ # - name: files-pv
+ # persistentVolumeClaim:
+ # claimName: files-pvc
containers:
- name: backend
image: {{ .Values.streampipes.registry }}/backend:{{ .Values.streampipes.version }}
imagePullPolicy: {{ .Values.pullPolicy }}
+ env:
+ - name: SP_PRIORITIZED_PROTOCOL
+ {{ if (eq .Values.deployment "lite") }}
+ value: "mqtt"
+ {{ else }}
+ value: "kafka"
+ {{ end }}
ports:
- containerPort: 8030
volumeMounts:
- mountPath: "/root/.streampipes"
name: backend-pv
- - mountPath: "/spImages"
- name: files-pv
+ # - mountPath: "/spImages"
+ # name: files-pv
diff --git a/k8s/templates/core/backend/backend-pvc.yaml b/k8s/templates/core/backend-pv-pvc.yaml
similarity index 65%
rename from k8s/templates/core/backend/backend-pvc.yaml
rename to k8s/templates/core/backend-pv-pvc.yaml
index a5776c1..724b87f 100644
--- a/k8s/templates/core/backend/backend-pvc.yaml
+++ b/k8s/templates/core/backend-pv-pvc.yaml
@@ -14,12 +14,29 @@
# limitations under the License.
apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: backend-pv
+spec:
+ storageClassName: local-storage-backend
+ capacity:
+ storage: 50Mi
+ accessModes:
+ - {{ .Values.persistentVolumeAccessModes }}
+ persistentVolumeReclaimPolicy: {{ .Values.persistentVolumeReclaimPolicy }}
+ hostPath:
+ path: {{ .Values.hostPath }}/backend
+---
+apiVersion: v1
kind: PersistentVolumeClaim
metadata:
+ labels:
+ app: backend
name: backend-pvc
spec:
+ storageClassName: local-storage-backend
accessModes:
- - ReadWriteOnce
+ - {{ .Values.persistentVolumeAccessModes }}
resources:
requests:
- storage: 50Mi
+ storage: 50Mi
\ No newline at end of file
diff --git a/k8s/templates/core/backend/backend-service.yaml b/k8s/templates/core/backend-service.yaml
similarity index 97%
rename from k8s/templates/core/backend/backend-service.yaml
rename to k8s/templates/core/backend-service.yaml
index 4c5c222..24dc03c 100644
--- a/k8s/templates/core/backend/backend-service.yaml
+++ b/k8s/templates/core/backend-service.yaml
@@ -24,4 +24,4 @@ spec:
- name: main
protocol: TCP
port: 8030
- targetPort: 8030
+ targetPort: 8030
\ No newline at end of file
diff --git a/k8s/templates/core/ui/ui-ingress.yaml b/k8s/templates/core/ui-ingress.yaml
similarity index 97%
rename from k8s/templates/core/ui/ui-ingress.yaml
rename to k8s/templates/core/ui-ingress.yaml
index c59b690..c6cd0d2 100644
--- a/k8s/templates/core/ui/ui-ingress.yaml
+++ b/k8s/templates/core/ui-ingress.yaml
@@ -20,4 +20,4 @@ metadata:
spec:
backend:
serviceName: ui
- servicePort: 80
+ servicePort: 80
\ No newline at end of file
diff --git a/k8s/templates/core/ui/ui-service.yaml b/k8s/templates/core/ui-service.yaml
similarity index 97%
rename from k8s/templates/core/ui/ui-service.yaml
rename to k8s/templates/core/ui-service.yaml
index cc10932..f1e3bb7 100644
--- a/k8s/templates/core/ui/ui-service.yaml
+++ b/k8s/templates/core/ui-service.yaml
@@ -25,4 +25,4 @@ spec:
- name: main
protocol: TCP
port: 80
- targetPort: 80
+ targetPort: 80
\ No newline at end of file
diff --git a/k8s/templates/core/ui/ui-deployment.yaml b/k8s/templates/core/ui.yaml
similarity index 68%
rename from k8s/templates/core/ui/ui-deployment.yaml
rename to k8s/templates/core/ui.yaml
index 840bcac..09a457e 100644
--- a/k8s/templates/core/ui/ui-deployment.yaml
+++ b/k8s/templates/core/ui.yaml
@@ -27,19 +27,6 @@ spec:
labels:
app: ui
spec:
- initContainers:
- - name: init-wait-backend
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 backend 8030 && exit 0 || sleep 3; done; exit 1"]
- - name: init-wait-consul
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 consul 8600 && exit 0 || sleep 3; done; exit 1"]
- - name: init-wait-couchdb
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 couchdb 5984 && exit 0 || sleep 3; done; exit 1"]
- - name: init-wait-activemq
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 activemq 61614 && exit 0 || sleep 3; done; exit 1"]
containers:
- name: ui
image: {{ .Values.streampipes.registry }}/ui:{{ .Values.streampipes.version }}
diff --git a/k8s/templates/extensions/connect-adapters/connect-worker-deployment.yaml b/k8s/templates/extensions/connect-adapters/connect-worker-deployment.yaml
index d3e87c4..022c78d 100644
--- a/k8s/templates/extensions/connect-adapters/connect-worker-deployment.yaml
+++ b/k8s/templates/extensions/connect-adapters/connect-worker-deployment.yaml
@@ -26,30 +26,10 @@ spec:
metadata:
labels:
app: connect-worker
- spec:
- initContainers:
- - name: init-wait-consul
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 consul 8600 && exit 0 || sleep 3; done; exit 1"]
- - name: init-wait-couchdb
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 couchdb 5984 && exit 0 || sleep 3; done; exit 1"]
- - name: init-wait-connect-worker
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 connect-master 8099 && exit 0 || sleep 3; done; exit 1"]
- - name: init-wait-kafka
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 kafka 9092 && exit 0 || sleep 3; done; exit 1"]
+ spec:
containers:
- name: connect-worker
image: {{ .Values.streampipes.registry }}/connect-worker:{{ .Values.streampipes.version }}
imagePullPolicy: {{ .Values.pullPolicy }}
- env:
- - name: SP_BACKEND_HOST
- value: backend
- - name: SP_CONNECT_CONTAINER_WORKER_HOST
- value: connect-worker
- - name: SP_KAFKA_HOST
- value: kafka
ports:
- containerPort: 8098
diff --git a/k8s/templates/extensions/pipeline-elements/files-pvc.yaml b/k8s/templates/extensions/pipeline-elements/files-pvc.yaml
index 0c117f9..bd6a088 100644
--- a/k8s/templates/extensions/pipeline-elements/files-pvc.yaml
+++ b/k8s/templates/extensions/pipeline-elements/files-pvc.yaml
@@ -13,14 +13,30 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: files-pv
+spec:
+ storageClassName: local-storage-files
+ capacity:
+ storage: 250Mi
+ accessModes:
+ - {{ .Values.persistentVolumeAccessModes }}
+ persistentVolumeReclaimPolicy: {{ .Values.persistentVolumeReclaimPolicy }}
+ hostPath:
+ path: {{ .Values.hostPath }}/files
+---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
+ labels:
+ app: pipeline-elements-all-jvm
name: files-pvc
spec:
+ storageClassName: local-storage-files
accessModes:
- - ReadWriteOnce
+ - {{ .Values.persistentVolumeAccessModes }}
resources:
requests:
- storage: 256Mi
+ storage: 250Mi
\ No newline at end of file
diff --git a/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-flink-deployment.yaml b/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-flink-deployment.yaml
index 2e8372f..c9ab590 100644
--- a/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-flink-deployment.yaml
+++ b/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-flink-deployment.yaml
@@ -28,13 +28,6 @@ spec:
labels:
app: pipeline-elements-all-flink
spec:
- initContainers:
- - name: init-wait-consul
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 consul 8600 && exit 0 || sleep 3; done; exit 1"]
- - name: init-wait-jobmanager
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 jobmanager 8081 && exit 0 || sleep 3; done; exit 1"]
containers:
- name: pipeline-elements-all-flink
image: {{ .Values.streampipes.registry }}/pipeline-elements-all-flink:{{ .Values.streampipes.version }}
diff --git a/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-jvm-deployment.yaml b/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-jvm-deployment.yaml
index f25b980..6508e37 100644
--- a/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-jvm-deployment.yaml
+++ b/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-jvm-deployment.yaml
@@ -27,11 +27,6 @@ spec:
labels:
app: pipeline-elements-all-jvm
spec:
- initContainers:
- - name: init-wait-consul
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 consul 8600 && exit 0 || sleep 3; done; exit 1"]
- spec:
volumes:
- name: files-pv
persistentVolumeClaim:
diff --git a/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-jvm-service.yaml b/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-jvm-service.yaml
index 4072603..1b1b01c 100644
--- a/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-jvm-service.yaml
+++ b/k8s/templates/extensions/pipeline-elements/pipeline-elements-all-jvm-service.yaml
@@ -26,4 +26,4 @@ spec:
- name: main
protocol: TCP
port: 8090
- targetPort: 8090
+ targetPort: 8090
\ No newline at end of file
diff --git a/k8s/templates/extensions/pipeline-elements/sources-vehicle-simulator-deployment.yaml b/k8s/templates/extensions/pipeline-elements/sources-vehicle-simulator-deployment.yaml
index acf6412..083926d 100644
--- a/k8s/templates/extensions/pipeline-elements/sources-vehicle-simulator-deployment.yaml
+++ b/k8s/templates/extensions/pipeline-elements/sources-vehicle-simulator-deployment.yaml
@@ -29,9 +29,6 @@ spec:
app: sources-vehicle-simulator
spec:
initContainers:
- - name: init-wait-consul
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 consul 8600 && exit 0 || sleep 3; done; exit 1"]
- name: init-wait-kafka
image: alpine
command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 kafka 9092 && exit 0 || sleep 3; done; exit 1"]
diff --git a/k8s/templates/extensions/pipeline-elements/sources-watertank-simulator-deployment.yaml b/k8s/templates/extensions/pipeline-elements/sources-watertank-simulator-deployment.yaml
index bc3afb0..e0cf433 100644
--- a/k8s/templates/extensions/pipeline-elements/sources-watertank-simulator-deployment.yaml
+++ b/k8s/templates/extensions/pipeline-elements/sources-watertank-simulator-deployment.yaml
@@ -13,6 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+{{- if (eq .Values.deployment "full") }}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -28,9 +29,6 @@ spec:
app: sources-watertank-simulator
spec:
initContainers:
- - name: init-wait-consul
- image: alpine
- command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 consul 8600 && exit 0 || sleep 3; done; exit 1"]
- name: init-wait-kafka
image: alpine
command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 kafka 9092 && exit 0 || sleep 3; done; exit 1"]
@@ -40,3 +38,4 @@ spec:
imagePullPolicy: {{ .Values.pullPolicy }}
ports:
- containerPort: 8090
+{{- end }}
\ No newline at end of file
diff --git a/k8s/templates/extensions/pipeline-elements/sources-watertank-simulator-service.yaml b/k8s/templates/extensions/pipeline-elements/sources-watertank-simulator-service.yaml
deleted file mode 100644
index 32147c7..0000000
--- a/k8s/templates/extensions/pipeline-elements/sources-watertank-simulator-service.yaml
+++ /dev/null
@@ -1,29 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: v1
-kind: Service
-metadata:
- name: sources-watertank-simulator
- labels:
- name: sources-watertank-simulator
-spec:
- selector:
- app: sources-watertank-simulator
- ports:
- - name: main
- protocol: TCP
- port: 8090
- targetPort: 8090
diff --git a/k8s/templates/external/activemq/activemq-deployment.yaml b/k8s/templates/external/activemq/activemq-deployment.yaml
index 0f6326e..90ce56d 100644
--- a/k8s/templates/external/activemq/activemq-deployment.yaml
+++ b/k8s/templates/external/activemq/activemq-deployment.yaml
@@ -13,6 +13,32 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+---
+apiVersion: v1
+kind: Service
+metadata:
+ name: activemq
+spec:
+ selector:
+ app: activemq
+ ports:
+ - name: main
+ protocol: TCP
+ port: 61616
+ targetPort: 61616
+ - name: websocket
+ protocol: TCP
+ port: 61614
+ targetPort: 61614
+ - name: ui
+ protocol: TCP
+ port: 8161
+ targetPort: 8161
+ - name: mqtt
+ protocol: TCP
+ port: 1883
+ targetPort: 1883
+---
apiVersion: apps/v1
kind: Deployment
metadata:
diff --git a/k8s/templates/external/activemq/activemq-service.yaml b/k8s/templates/external/activemq/activemq-service.yaml
index 745179a..8b8c874 100644
--- a/k8s/templates/external/activemq/activemq-service.yaml
+++ b/k8s/templates/external/activemq/activemq-service.yaml
@@ -36,4 +36,4 @@ spec:
- name: mqtt
protocol: TCP
port: 1883
- targetPort: 1883
+ targetPort: 1883
\ No newline at end of file
diff --git a/k8s/templates/external/consul/consul-deployment.yaml b/k8s/templates/external/consul/consul-deployment.yaml
index bac4479..b80fc11 100644
--- a/k8s/templates/external/consul/consul-deployment.yaml
+++ b/k8s/templates/external/consul/consul-deployment.yaml
@@ -35,6 +35,26 @@ spec:
- name: consul
image: fogsyio/consul:{{ .Values.external.consulVersion }}
imagePullPolicy: {{ .Values.pullPolicy }}
+ command:
+ - "/bin/sh"
+ - "-ec"
+ - |
+ exec /bin/consul agent \
+ -server \
+ -bind=0.0.0.0 \
+ -bootstrap-expect=1 \
+ -data-dir=/consul/data \
+ -node=consule-one \
+ -client=0.0.0.0 \
+ -enable-script-checks=true \
+ -ui
+ env:
+ - name: CONSUL_BIND_INTERFACE
+ value: "eth0"
+ - name: CONSUL_HTTP_ADDR
+ value: "0.0.0.0"
+ - name: CONSUL_LOCAL_CONFIG
+ value: "{\"disable_update_check\": true}"
ports:
- containerPort: 8500
- containerPort: 8600
diff --git a/k8s/templates/external/consul/consul-pvc.yaml b/k8s/templates/external/consul/consul-pv-pvc.yaml
similarity index 65%
rename from k8s/templates/external/consul/consul-pvc.yaml
rename to k8s/templates/external/consul/consul-pv-pvc.yaml
index 829251a..02cba15 100644
--- a/k8s/templates/external/consul/consul-pvc.yaml
+++ b/k8s/templates/external/consul/consul-pv-pvc.yaml
@@ -14,12 +14,29 @@
# limitations under the License.
apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: consul-pv
+spec:
+ storageClassName: local-storage-consul
+ capacity:
+ storage: 20M
+ accessModes:
+ - {{ .Values.persistentVolumeAccessModes }}
+ persistentVolumeReclaimPolicy: {{ .Values.persistentVolumeReclaimPolicy }}
+ hostPath:
+ path: {{ .Values.hostPath }}/consul
+---
+apiVersion: v1
kind: PersistentVolumeClaim
metadata:
+ labels:
+ app: consul
name: consul-pvc
spec:
+ storageClassName: local-storage-consul
accessModes:
- - ReadWriteOnce
+ - {{ .Values.persistentVolumeAccessModes }}
resources:
requests:
- storage: 1Gi
\ No newline at end of file
+ storage: 20M
\ No newline at end of file
diff --git a/k8s/templates/external/couchdb/couchdb-pvc.yaml b/k8s/templates/external/couchdb/couchdb-pv-pvc.yaml
similarity index 65%
rename from k8s/templates/external/couchdb/couchdb-pvc.yaml
rename to k8s/templates/external/couchdb/couchdb-pv-pvc.yaml
index b2f7d17..32861e0 100644
--- a/k8s/templates/external/couchdb/couchdb-pvc.yaml
+++ b/k8s/templates/external/couchdb/couchdb-pv-pvc.yaml
@@ -14,12 +14,29 @@
# limitations under the License.
apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: couchdb-pv
+spec:
+ storageClassName: local-storage-couchdb
+ capacity:
+ storage: 50M
+ accessModes:
+ - {{ .Values.persistentVolumeAccessModes }}
+ persistentVolumeReclaimPolicy: {{ .Values.persistentVolumeReclaimPolicy }}
+ hostPath:
+ path: {{ .Values.hostPath }}/couchdb
+---
+apiVersion: v1
kind: PersistentVolumeClaim
metadata:
+ labels:
+ app: couchdb
name: couchdb-pvc
spec:
+ storageClassName: local-storage-couchdb
accessModes:
- - ReadWriteOnce
+ - {{ .Values.persistentVolumeAccessModes }}
resources:
requests:
- storage: 1Gi
\ No newline at end of file
+ storage: 50M
\ No newline at end of file
diff --git a/k8s/templates/external/flink/jobmanager-service.yaml b/k8s/templates/external/flink/jobmanager-service.yaml
index 2bf1e56..6c7ec77 100644
--- a/k8s/templates/external/flink/jobmanager-service.yaml
+++ b/k8s/templates/external/flink/jobmanager-service.yaml
@@ -30,5 +30,5 @@ spec:
targetPort: 6123
- name: blob-server
port: 6124
- targetPort: 6124
+ targetPort: 6124
{{- end }}
diff --git a/k8s/templates/external/flink/taskmanager-service.yaml b/k8s/templates/external/flink/taskmanager-service.yaml
deleted file mode 100644
index c76bfe7..0000000
--- a/k8s/templates/external/flink/taskmanager-service.yaml
+++ /dev/null
@@ -1,29 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-{{- if eq .Values.deployment "full" }}
-apiVersion: v1
-kind: Service
-metadata:
- name: taskmanager
-spec:
- selector:
- app: taskmanager
- ports:
- - name: main
- protocol: TCP
- port: 8099
- targetPort: 8099
-{{- end }}
diff --git a/k8s/templates/external/influxdb/influxdb-pvc.yaml b/k8s/templates/external/influxdb/influxdb-pvc.yaml
index 57f0bd3..fbcb64a 100644
--- a/k8s/templates/external/influxdb/influxdb-pvc.yaml
+++ b/k8s/templates/external/influxdb/influxdb-pvc.yaml
@@ -14,12 +14,29 @@
# limitations under the License.
apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: influx-pv
+spec:
+ storageClassName: local-storage-influx
+ capacity:
+ storage: 250Mi
+ accessModes:
+ - {{ .Values.persistentVolumeAccessModes }}
+ persistentVolumeReclaimPolicy: {{ .Values.persistentVolumeReclaimPolicy }}
+ hostPath:
+ path: {{ .Values.hostPath }}/influxdb
+---
+apiVersion: v1
kind: PersistentVolumeClaim
metadata:
+ labels:
+ app: influxdb
name: influxdb-pvc
spec:
+ storageClassName: local-storage-influx
accessModes:
- - ReadWriteOnce
+ - {{ .Values.persistentVolumeAccessModes }}
resources:
requests:
- storage: 5Gi
+ storage: 250Mi
diff --git a/k8s/templates/external/kafka/kafka-deployment.yaml b/k8s/templates/external/kafka/kafka-deployment.yaml
index 4d45116..eb844d7 100644
--- a/k8s/templates/external/kafka/kafka-deployment.yaml
+++ b/k8s/templates/external/kafka/kafka-deployment.yaml
@@ -13,6 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+{{- if (eq .Values.deployment "full") }}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -65,3 +66,4 @@ spec:
value: "5000012"
- name: KAFKA_REPLICA_FETCH_MAX_BYTES
value: "10000000"
+{{- end }}
\ No newline at end of file
diff --git a/k8s/templates/external/kafka/kafka-pvc.yaml b/k8s/templates/external/kafka/kafka-pvc.yaml
index 48281a3..3bda2e2 100644
--- a/k8s/templates/external/kafka/kafka-pvc.yaml
+++ b/k8s/templates/external/kafka/kafka-pvc.yaml
@@ -13,13 +13,32 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+{{- if (eq .Values.deployment "full") }}
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: kafka-pv
+spec:
+ storageClassName: local-storage-kafka
+ capacity:
+ storage: 50M
+ accessModes:
+ - {{ .Values.persistentVolumeAccessModes }}
+ persistentVolumeReclaimPolicy: {{ .Values.persistentVolumeReclaimPolicy }}
+ hostPath:
+ path: {{ .Values.hostPath }}/kafka
+---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
+ labels:
+ app: kafka
name: kafka-pvc
spec:
+ storageClassName: local-storage-kafka
accessModes:
- - ReadWriteOnce
+ - {{ .Values.persistentVolumeAccessModes }}
resources:
requests:
- storage: 5Gi
+ storage: 50M
+{{- end }}
\ No newline at end of file
diff --git a/k8s/templates/external/kafka/kafka-service.yaml b/k8s/templates/external/kafka/kafka-service.yaml
index 1e1c265..1b75c43 100644
--- a/k8s/templates/external/kafka/kafka-service.yaml
+++ b/k8s/templates/external/kafka/kafka-service.yaml
@@ -13,6 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+{{- if (eq .Values.deployment "full") }}
apiVersion: v1
kind: Service
metadata:
@@ -26,3 +27,4 @@ spec:
- name: main
protocol: TCP
port: 9092
+{{- end }}
\ No newline at end of file
diff --git a/k8s/templates/external/zookeeper/zookeeper-deployment.yaml b/k8s/templates/external/zookeeper/zookeeper-deployment.yaml
index 756ae4e..6debfc6 100644
--- a/k8s/templates/external/zookeeper/zookeeper-deployment.yaml
+++ b/k8s/templates/external/zookeeper/zookeeper-deployment.yaml
@@ -13,6 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+{{- if (eq .Values.deployment "full") }}
apiVersion: apps/v1
kind: Deployment
metadata:
@@ -42,3 +43,4 @@ spec:
volumeMounts:
- mountPath: "/opt/zookeeper-{{ .Values.external.zookeeperVersion }}/data"
name: zookeeper-pv
+{{- end }}
\ No newline at end of file
diff --git a/k8s/templates/external/zookeeper/zookeeper-pvc.yaml b/k8s/templates/external/zookeeper/zookeeper-pvc.yaml
index 4170f8e..cbd9b45 100644
--- a/k8s/templates/external/zookeeper/zookeeper-pvc.yaml
+++ b/k8s/templates/external/zookeeper/zookeeper-pvc.yaml
@@ -13,13 +13,32 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+{{- if (eq .Values.deployment "full") }}
+apiVersion: v1
+kind: PersistentVolume
+metadata:
+ name: zookeeper-pv
+spec:
+ storageClassName: local-storage-zookeeper
+ capacity:
+ storage: 20M
+ accessModes:
+ - {{ .Values.persistentVolumeAccessModes }}
+ persistentVolumeReclaimPolicy: {{ .Values.persistentVolumeReclaimPolicy }}
+ hostPath:
+ path: {{ .Values.hostPath }}/zookeeper
+---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
+ labels:
+ app: zookeeper
name: zookeeper-pvc
spec:
+ storageClassName: local-storage-zookeeper
accessModes:
- - ReadWriteOnce
+ - {{ .Values.persistentVolumeAccessModes }}
resources:
requests:
- storage: 1Gi
+ storage: 20M
+{{- end }}
\ No newline at end of file
diff --git a/k8s/templates/external/zookeeper/zookeeper-service.yaml b/k8s/templates/external/zookeeper/zookeeper-service.yaml
index 7f45877..cb10589 100644
--- a/k8s/templates/external/zookeeper/zookeeper-service.yaml
+++ b/k8s/templates/external/zookeeper/zookeeper-service.yaml
@@ -13,6 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+{{- if (eq .Values.deployment "full") }}
apiVersion: v1
kind: Service
metadata:
@@ -24,4 +25,5 @@ spec:
- name: main
protocol: TCP
port: 2181
- targetPort: 2181
\ No newline at end of file
+ targetPort: 2181
+{{- end }}
\ No newline at end of file
diff --git a/k8s/values.yaml b/k8s/values.yaml
index 899696f..bb2e145 100644
--- a/k8s/values.yaml
+++ b/k8s/values.yaml
@@ -16,6 +16,8 @@
# lite or full (default: lite)
deployment: lite
pullPolicy: "Always"
+persistentVolumeReclaimPolicy: "Retain"
+persistentVolumeAccessModes: "ReadWriteOnce"
streampipes:
version: "0.68.0-SNAPSHOT"
@@ -29,7 +31,8 @@ external:
kafkaVersion: 2.2.0
zookeeperVersion: 3.4.13
influxdbVersion: 1.7
- iotdbVersion: latest
- kibanaVersion: 6.2.3
- elasticsearchVersion: 6.2.3
- redisVersion: latest
+
+# NOTE: Local testing only!
+# minikube start --mount-string ${HOME}/streampipes-k8s:/streampipes-k8s --mount --memory=4g --cpus=4
+# Specify minikube directory:
+hostPath: "/streampipes-k8s"
\ No newline at end of file
[incubator-streampipes-installer] 02/02: Add timescaledb test
instance to cli
Posted by wi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wiener pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-streampipes-installer.git
commit 3dac44ee431b73c3a899b99a5df1eeaa0ec33efe
Author: Patrick Wiener <wi...@fzi.de>
AuthorDate: Thu Dec 10 09:36:09 2020 +0100
Add timescaledb test instance to cli
---
.../standalone/timescaledb/docker-compose.dev.yml | 20 +++++++++++
.../standalone/timescaledb/docker-compose.yml | 39 ++++++++++++++++++++++
2 files changed, 59 insertions(+)
diff --git a/cli/deploy/standalone/timescaledb/docker-compose.dev.yml b/cli/deploy/standalone/timescaledb/docker-compose.dev.yml
new file mode 100644
index 0000000..d483d1c
--- /dev/null
+++ b/cli/deploy/standalone/timescaledb/docker-compose.dev.yml
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3.4"
+services:
+ timescaledb:
+ ports:
+ - 5432:5432
diff --git a/cli/deploy/standalone/timescaledb/docker-compose.yml b/cli/deploy/standalone/timescaledb/docker-compose.yml
new file mode 100644
index 0000000..7e51e69
--- /dev/null
+++ b/cli/deploy/standalone/timescaledb/docker-compose.yml
@@ -0,0 +1,39 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3.4"
+services:
+ timescaledb:
+ image: timescale/timescaledb:1.7.4-pg12
+ ports:
+ - "5432:5432"
+ volumes:
+ - timescaledb:/var/lib/postgresql/data
+ environment:
+ POSTGRES_PASSWORD: timescalepw
+ logging:
+ driver: "json-file"
+ options:
+ max-size: "1m"
+ max-file: "1"
+ networks:
+ spnet:
+
+volumes:
+ timescaledb:
+
+networks:
+ spnet:
+ external: true