You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/07/01 22:46:46 UTC
(texera) branch gh-readonly-queue/main/pr-5932-a1a7eb031628507c30ecd0eadb9b744b14e52c71 created (now 322babf620)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-5932-a1a7eb031628507c30ecd0eadb9b744b14e52c71
in repository https://gitbox.apache.org/repos/asf/texera.git
at 322babf620 feat(k8s): make app-tier object storage configurable via storage.s3 (#5932)
This branch includes the following new commits:
new 322babf620 feat(k8s): make app-tier object storage configurable via storage.s3 (#5932)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(texera) 01/01: feat(k8s): make app-tier object storage configurable via storage.s3 (#5932)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-5932-a1a7eb031628507c30ecd0eadb9b744b14e52c71
in repository https://gitbox.apache.org/repos/asf/texera.git
commit 322babf6205cf72277d13a4f2d16c5954a4dfbd9
Author: ali risheh <al...@gmail.com>
AuthorDate: Wed Jul 1 15:46:27 2026 -0700
feat(k8s): make app-tier object storage configurable via storage.s3 (#5932)
### What changes were proposed in this PR?
This makes the **application tier's** object-storage target
configurable, as the first step toward supporting an external S3 store
alongside the in-cluster MinIO. It is **non-breaking**: the default
(on-prem / in-cluster MinIO) install renders an identical set of
resources.
Today `file-service` and `workflow-computing-unit-manager` hardcode the
S3 endpoint and credentials to the in-cluster MinIO Service (`{{
.Release.Name }}-minio`) and its auto-generated Secret. This PR routes
both through a new `storage.s3` values block:
- **`values.yaml`** — new `storage.s3` block: `endpoint`, `region`,
`existingSecret`, `accessKeyId`, `secretAccessKey`. All default to
empty.
- **`templates/base/_helpers.tpl`** (new) — helpers that resolve the S3
endpoint and the credentials Secret name/keys. When
`storage.s3.endpoint` is empty they fall back to the in-cluster MinIO
Service and its `{{ .Release.Name }}-minio` Secret (keys
`root-user`/`root-password`), so the default render is unchanged.
- **`templates/aws/s3-credentials-secret.yaml`** (new) — a `{{
.Release.Name }}-s3-credentials` Secret, rendered **only** when an
external `endpoint` is set and no `existingSecret` is supplied. Renders
nothing on the default install.
- **`file-service` / `workflow-computing-unit-manager` deployments** —
the `STORAGE_S3_*` env now comes from the helpers; `STORAGE_S3_REGION`
is added only in external mode.
How it behaves:
- **Default (no config):** services point at the in-cluster MinIO
exactly as before.
- **External S3:** set `storage.s3.endpoint` + `region` + credentials
(or `existingSecret`) → both services use that store; the chart
materializes the credentials Secret unless you bring your own.
**Out of scope (intentionally deferred to keep this PR small and
atomic):** LakeFS blockstore + Lakekeeper warehouse external-S3 wiring,
the `minio.enabled` switch to drop the in-cluster MinIO entirely, and a
`values-aws.yaml` example overlay. Those are the LakeFS/Lakekeeper half
of "make object storage pluggable" and will land as a follow-up.
### Any related issues, documentation, discussions?
Closes #5931 (app-tier `storage.s3` task).
Part of #5891 — unify AWS (EKS) and on-premise Kubernetes deployment
under `bin/k8s` (parent feature).
Follows #5757 (Helm template reorg) and the design discussion in #5641.
### How was this PR tested?
Verified the default install is unchanged and the external path renders
correctly:
1. **No-op render proof:** `helm template texera bin/k8s` on this branch
vs on `main` renders the **same 102 resources** — identical after
ignoring comments. The only textual artifact is one empty `---` document
from the gated-off `s3-credentials-secret.yaml` (its `# license` header
is emitted by Helm even though the `{{- if }}` body is empty); it
produces no Kubernetes object. `helm lint` passes.
2. **External S3 render:** `helm template ... --set
storage.s3.endpoint=https://s3.us-west-2.amazonaws.com --set
storage.s3.accessKeyId=… --set storage.s3.secretAccessKey=…` renders the
`*-s3-credentials` Secret, repoints both deployments'
`STORAGE_S3_ENDPOINT`/credentials at it (keys
`access-key-id`/`secret-access-key`), and adds `STORAGE_S3_REGION`.
3. **Bring-your-own Secret:** with `--set
storage.s3.existingSecret=my-creds`, the deployments reference
`my-creds` and the chart generates **no** Secret.
4. `helm lint bin/k8s` passes for both the default and the external
value sets.
5. `helm install` locally with Minikube and test creating dataset and
running workflows
No unit tests were added — the change is limited to Helm chart
values/templates, validated by the render diff and `helm lint` above.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Co-authored-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
bin/k8s/templates/aws/s3-credentials-secret.yaml | 32 +++++++++++++
bin/k8s/templates/base/_helpers.tpl | 56 ++++++++++++++++++++++
.../base/file-service/file-service-deployment.yaml | 14 ++++--
...workflow-computing-unit-manager-deployment.yaml | 14 ++++--
bin/k8s/values.yaml | 12 +++++
5 files changed, 118 insertions(+), 10 deletions(-)
diff --git a/bin/k8s/templates/aws/s3-credentials-secret.yaml b/bin/k8s/templates/aws/s3-credentials-secret.yaml
new file mode 100644
index 0000000000..8506115167
--- /dev/null
+++ b/bin/k8s/templates/aws/s3-credentials-secret.yaml
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Credentials Secret for an external S3 store. Rendered only when an external
+# endpoint is configured (storage.s3.endpoint) and the deployer has not supplied
+# their own Secret (storage.s3.existingSecret). On the default in-cluster MinIO
+# install this renders nothing -- the services use MinIO's own Secret instead.
+{{- if and .Values.storage.s3.endpoint (not .Values.storage.s3.existingSecret) }}
+apiVersion: v1
+kind: Secret
+metadata:
+ name: {{ .Release.Name }}-s3-credentials
+ namespace: {{ .Release.Namespace }}
+type: Opaque
+stringData:
+ access-key-id: {{ .Values.storage.s3.accessKeyId | quote }}
+ secret-access-key: {{ .Values.storage.s3.secretAccessKey | quote }}
+{{- end }}
diff --git a/bin/k8s/templates/base/_helpers.tpl b/bin/k8s/templates/base/_helpers.tpl
new file mode 100644
index 0000000000..e044b7285a
--- /dev/null
+++ b/bin/k8s/templates/base/_helpers.tpl
@@ -0,0 +1,56 @@
+{{/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/}}
+
+{{/*
+Object-storage (S3) resolution helpers.
+
+When storage.s3.endpoint is set the services talk to that external
+S3-compatible store (credentials come from storage.s3.existingSecret, or a
+chart-generated "<release>-s3-credentials" Secret). When it is empty the
+services fall back to the in-cluster MinIO Service and its auto-generated
+"<release>-minio" Secret, so the default install is unchanged.
+*/}}
+
+{{/* S3 endpoint URL. */}}
+{{- define "texera.s3.endpoint" -}}
+{{- if .Values.storage.s3.endpoint -}}
+{{- .Values.storage.s3.endpoint -}}
+{{- else -}}
+{{- printf "http://%s-minio:9000" .Release.Name -}}
+{{- end -}}
+{{- end -}}
+
+{{/* Name of the Secret holding the S3 credentials. */}}
+{{- define "texera.s3.secretName" -}}
+{{- if .Values.storage.s3.endpoint -}}
+{{- .Values.storage.s3.existingSecret | default (printf "%s-s3-credentials" .Release.Name) -}}
+{{- else -}}
+{{- printf "%s-minio" .Release.Name -}}
+{{- end -}}
+{{- end -}}
+
+{{/* Secret data key for the S3 access key id. */}}
+{{- define "texera.s3.accessKeyIdKey" -}}
+{{- if .Values.storage.s3.endpoint -}}access-key-id{{- else -}}root-user{{- end -}}
+{{- end -}}
+
+{{/* Secret data key for the S3 secret access key. */}}
+{{- define "texera.s3.secretAccessKeyKey" -}}
+{{- if .Values.storage.s3.endpoint -}}secret-access-key{{- else -}}root-password{{- end -}}
+{{- end -}}
diff --git a/bin/k8s/templates/base/file-service/file-service-deployment.yaml b/bin/k8s/templates/base/file-service/file-service-deployment.yaml
index 6c9a4041f9..6a9190bc6c 100644
--- a/bin/k8s/templates/base/file-service/file-service-deployment.yaml
+++ b/bin/k8s/templates/base/file-service/file-service-deployment.yaml
@@ -41,17 +41,21 @@ spec:
env:
# LakeFS & S3 Access
- name: STORAGE_S3_ENDPOINT
- value: http://{{ .Release.Name }}-minio:9000
+ value: {{ include "texera.s3.endpoint" . }}
+ {{- if .Values.storage.s3.endpoint }}
+ - name: STORAGE_S3_REGION
+ value: {{ .Values.storage.s3.region | quote }}
+ {{- end }}
- name: STORAGE_S3_AUTH_USERNAME
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-user
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.accessKeyIdKey" . }}
- name: STORAGE_S3_AUTH_PASSWORD
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-password
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.secretAccessKeyKey" . }}
- name: STORAGE_LAKEFS_ENDPOINT
value: http://{{ .Release.Name }}-lakefs:8000/api/v1
- name: STORAGE_LAKEFS_AUTH_USERNAME
diff --git a/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml b/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml
index 7a0185cd46..a911845041 100644
--- a/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml
+++ b/bin/k8s/templates/base/workflow-computing-unit-manager/workflow-computing-unit-manager-deployment.yaml
@@ -81,17 +81,21 @@ spec:
value: http://{{ .Values.fileService.name }}-svc:9092/api/dataset/did/upload
# S3 Access (for R UDF large binary support)
- name: STORAGE_S3_ENDPOINT
- value: http://{{ .Release.Name }}-minio:9000
+ value: {{ include "texera.s3.endpoint" . }}
+ {{- if .Values.storage.s3.endpoint }}
+ - name: STORAGE_S3_REGION
+ value: {{ .Values.storage.s3.region | quote }}
+ {{- end }}
- name: STORAGE_S3_AUTH_USERNAME
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-user
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.accessKeyIdKey" . }}
- name: STORAGE_S3_AUTH_PASSWORD
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-password
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.secretAccessKeyKey" . }}
# LakeFS Access (should be removed in production environment)
- name: STORAGE_LAKEFS_ENDPOINT
value: http://{{ .Release.Name }}-lakefs.{{ .Release.Namespace }}:8000/api/v1
diff --git a/bin/k8s/values.yaml b/bin/k8s/values.yaml
index 2974c27c88..64642b517a 100644
--- a/bin/k8s/values.yaml
+++ b/bin/k8s/values.yaml
@@ -65,6 +65,18 @@ postgresql:
initdb:
scriptsConfigMap: "postgresql-init-script"
+# Object storage (S3) used by the Texera services. Leave storage.s3.endpoint
+# empty to use the in-cluster MinIO (the `minio:` block below). Set it to an
+# S3-compatible endpoint URL -- together with region and credentials -- to
+# point the services at an external store (e.g. AWS S3) instead.
+storage:
+ s3:
+ endpoint: "" # "" => in-cluster MinIO; otherwise an S3 endpoint URL
+ region: "us-west-2"
+ existingSecret: "" # existing Secret with access-key-id / secret-access-key; "" => chart creates one
+ accessKeyId: ""
+ secretAccessKey: ""
+
minio:
mode: standalone
image: