You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/07/14 00:52:53 UTC
(texera) branch gh-readonly-queue/main/pr-6295-ff82fd82fd7229b7ea27a683af41454eebd74985 created (now 8770ab87ba)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-6295-ff82fd82fd7229b7ea27a683af41454eebd74985
in repository https://gitbox.apache.org/repos/asf/texera.git
at 8770ab87ba feat(k8s): external S3 for LakeFS & Lakekeeper, MinIO optional (#6295)
This branch includes the following new commits:
new 8770ab87ba feat(k8s): external S3 for LakeFS & Lakekeeper, MinIO optional (#6295)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(texera) 01/01: feat(k8s): external S3 for LakeFS & Lakekeeper, MinIO optional (#6295)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6295-ff82fd82fd7229b7ea27a683af41454eebd74985
in repository https://gitbox.apache.org/repos/asf/texera.git
commit 8770ab87bade26e452b87a3979e4c78c0ea0bdd5
Author: ali risheh <al...@gmail.com>
AuthorDate: Mon Jul 13 17:52:32 2026 -0700
feat(k8s): external S3 for LakeFS & Lakekeeper, MinIO optional (#6295)
### What changes were proposed in this PR?
This completes the "make object storage pluggable" work by wiring the
**data tier** (LakeFS blockstore + Lakekeeper Iceberg warehouse) to an
external S3 store and adding a single switch to drop the in-cluster
MinIO entirely. #5932 did the **application tier** (`storage.s3`) and
explicitly deferred this half. It is **non-breaking**: the default
(on-prem / in-cluster MinIO) install renders the same set of resources.
- **`minio.enabled` switch** (`values.yaml`, default `true`) to turn off
the bundled MinIO subchart and point everything at external S3:
- `Chart.yaml` -- the `minio` dependency gets `condition:
minio.enabled`.
- `templates/on-prem/minio-persistence.yaml` -- the MinIO PV/PVC is
gated on `minio.enabled` as well as `minio.persistence.enabled`.
- `templates/base/external-names/external-names.yaml` -- the
`<release>-minio` `ExternalName` alias (exposed to the
computing-unit-pool namespace) is only rendered when MinIO is enabled;
with external S3 the CU pods reach the store directly via
`STORAGE_S3_ENDPOINT`.
- **Lakekeeper init job**
(`templates/base/lakekeeper/lakekeeper-init-job.yaml`):
- `STORAGE_S3_*` env now resolves through the shared `texera.s3.*`
helpers introduced in #5932 (endpoint + credentials Secret name/keys)
instead of hardcoding `<release>-minio`.
- New `lakekeeperInit.createBucket` (default `true`) -- when `false`,
the job skips `mc mb` and logs that it is using a pre-existing external
bucket. Needed because an external bucket already exists and its IAM
principal may lack `s3:CreateBucket`.
- New `lakekeeperInit.warehouse.flavor` -- `s3-compat` (MinIO/other;
emits `endpoint` + `path-style-access: true`) or native `aws` (real AWS
S3; omits both). New optional `lakekeeperInit.warehouse.keyPrefix` to
isolate the warehouse within a shared bucket.
- **`values.yaml`** -- documents `minio.enabled`, the LakeFS
external-blockstore override path, and the new `lakekeeperInit` keys.
- **`values-aws.yaml`** (new) -- a complete, copy-pasteable
all-external-S3 overlay: MinIO off, `storage.s3`
(endpoint/region/credentials or `existingSecret`), LakeFS region-only S3
blockstore with credentials injected via `extraEnvVars`, and Lakekeeper
`flavor: aws` + `createBucket: false`.
How it behaves:
- **Default (no config):** LakeFS and Lakekeeper use the in-cluster
MinIO exactly as before; the MinIO subchart, PV/PVC, and `ExternalName`
all render as today.
- **External S3 (`-f values-aws.yaml`):** no MinIO is deployed; the app
tier, LakeFS, and Lakekeeper all target the external S3 buckets, and the
Lakekeeper warehouse is registered with the native AWS flavor against a
pre-existing bucket.
### Any related issues, documentation, discussions?
Closes #6294 (LakeFS blockstore + Lakekeeper warehouse external-S3
task).
Part of #5891 -- unify AWS (EKS) and on-premise Kubernetes deployment
under `bin/k8s` (parent feature).
Follows #5932 (app-tier `storage.s3`) and #5757 (Helm template reorg).
### How was this PR tested?
**1. Render parity (default install unchanged).** `helm template texera
bin/k8s` on this branch vs on `main` renders the same set of resources;
`helm lint bin/k8s` passes. The new `{{- if }}` guards (MinIO
ExternalName, persistence, createBucket, flavor) all evaluate to their
prior behavior when `minio.enabled`/`createBucket` default to `true`.
**2. External-S3 render.** `helm template texera bin/k8s -f
bin/k8s/values-aws.yaml` deploys no MinIO subchart / PV / PVC /
ExternalName, routes the Lakekeeper init job's `STORAGE_S3_*` at the
external endpoint and credentials Secret, skips the `mc mb`
bucket-creation step, and registers the warehouse storage profile with
`"flavor": "aws"` (no `endpoint` / `path-style-access`). `helm lint`
passes for this value set.
**3. End-to-end on a live EKS cluster with real AWS S3.** Stood up an
isolated EKS cluster (`us-west-2`, its own AWS account, no MinIO) plus
two pre-created S3 buckets (LakeFS blockstore + Iceberg warehouse) and
an IAM user scoped to just those buckets with **no** `CreateBucket`
permission. Built all Texera service images from this branch, deployed
the chart with an external-S3 overlay derived from `values-aws.yaml`,
and exercised the full stack:
- LakeFS + Lakekeeper init jobs authenticated to external S3 and
initialized (Lakekeeper `aws` flavor, bucket-creation skipped against
the pre-existing warehouse bucket).
- Created a dataset -> objects written to the external LakeFS bucket;
dataset preview served via presigned S3 URLs (external buckets need a
CORS rule, noted below).
- Ran a workflow (CSV file scan) end to end -> **success, 100 rows
returned**, with the Iceberg result table physically materialized in the
external warehouse bucket (`metadata.json` + data objects confirmed via
the S3 API). This exercises the full external-S3 path: file-service
reads the CSV from LakeFS -> S3, the computing-unit pod reaches the
Lakekeeper REST catalog, and results are written to Iceberg -> S3 and
read back.
All test AWS resources (cluster, buckets, IAM user) were torn down
afterward.
Operational note surfaced by the live test (not a chart defect, worth
documenting): with a real external S3 store, dataset preview requires a
CORS rule on the buckets (`GET`/`HEAD`) because the presigned URLs are
served cross-origin from `*.s3.<region>.amazonaws.com`, whereas the
in-cluster MinIO was same-origin.
No unit tests were added -- the change is limited to Helm chart
values/templates, validated by the render diff, `helm lint`, and the
live end-to-end run above.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
bin/k8s/Chart.yaml | 1 +
.../base/external-names/external-names.yaml | 7 +-
.../base/lakekeeper/lakekeeper-init-job.yaml | 43 ++++++-----
bin/k8s/templates/on-prem/minio-persistence.yaml | 2 +-
bin/k8s/values-aws.yaml | 86 ++++++++++++++++++++++
bin/k8s/values.yaml | 17 +++++
6 files changed, 136 insertions(+), 20 deletions(-)
diff --git a/bin/k8s/Chart.yaml b/bin/k8s/Chart.yaml
index 548d9e531e..8cda7ee227 100644
--- a/bin/k8s/Chart.yaml
+++ b/bin/k8s/Chart.yaml
@@ -49,6 +49,7 @@ dependencies:
- name: minio
version: 15.0.7
repository: https://charts.bitnami.com/bitnami
+ condition: minio.enabled
- name: lakefs
version: 1.8.1
diff --git a/bin/k8s/templates/base/external-names/external-names.yaml b/bin/k8s/templates/base/external-names/external-names.yaml
index c79ce40f33..6879b8d7f7 100644
--- a/bin/k8s/templates/base/external-names/external-names.yaml
+++ b/bin/k8s/templates/base/external-names/external-names.yaml
@@ -73,13 +73,16 @@ to access services in the main namespace using the same service names.
"externalName" (printf "%s-svc.%s.svc.cluster.local" .Values.webserver.name $namespace)
) | nindent 0 }}
+{{- if .Values.minio.enabled }}
---
-{{/* MinIO ExternalName */}}
-{{- include "external-name-service" (dict
+{{/* MinIO ExternalName -- only when the in-cluster MinIO is enabled; with an
+ external S3 store the CU pods reach it directly via STORAGE_S3_ENDPOINT. */}}
+{{- include "external-name-service" (dict
"name" (printf "%s-minio" .Release.Name)
"namespace" $workflowComputingUnitPoolNamespace
"externalName" (printf "%s-minio.%s.svc.cluster.local" .Release.Name $namespace)
) | nindent 0 }}
+{{- end }}
---
{{/* Lakekeeper ExternalName */}}
diff --git a/bin/k8s/templates/base/lakekeeper/lakekeeper-init-job.yaml b/bin/k8s/templates/base/lakekeeper/lakekeeper-init-job.yaml
index 0a3540b605..d1adafcd92 100644
--- a/bin/k8s/templates/base/lakekeeper/lakekeeper-init-job.yaml
+++ b/bin/k8s/templates/base/lakekeeper/lakekeeper-init-job.yaml
@@ -16,6 +16,9 @@
# under the License.
{{- if .Values.lakekeeperInit.enabled }}
+{{- if not (has .Values.lakekeeperInit.warehouse.flavor (list "s3-compat" "aws")) }}
+{{- fail (printf "lakekeeperInit.warehouse.flavor must be \"s3-compat\" or \"aws\", got %q" .Values.lakekeeperInit.warehouse.flavor) }}
+{{- end }}
apiVersion: batch/v1
kind: Job
metadata:
@@ -33,17 +36,17 @@ spec:
image: alpine:3.19
env:
- name: STORAGE_S3_ENDPOINT
- value: http://{{ .Release.Name }}-minio:9000
+ value: {{ include "texera.s3.endpoint" . }}
- name: STORAGE_S3_AUTH_USERNAME
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-user
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.accessKeyIdKey" . }}
- name: STORAGE_S3_AUTH_PASSWORD
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-password
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.secretAccessKeyKey" . }}
- name: STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET
value: {{ .Values.lakekeeperInit.warehouse.s3Bucket | quote }}
- name: STORAGE_ICEBERG_CATALOG_REST_REGION
@@ -62,9 +65,7 @@ spec:
- |
set -e
- apk add --no-cache curl ca-certificates wget
- wget -q https://dl.min.io/client/mc/release/linux-amd64/mc -O /usr/local/bin/mc
- chmod +x /usr/local/bin/mc
+ apk add --no-cache curl ca-certificates
check_status() {
if [ "$1" -ge 200 ] && [ "$1" -lt 300 ]; then
@@ -85,14 +86,17 @@ spec:
sleep 3
done
- echo "Step 1: Initializing MinIO bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}'..."
- mc alias set minio "${STORAGE_S3_ENDPOINT}" "${STORAGE_S3_AUTH_USERNAME}" "${STORAGE_S3_AUTH_PASSWORD}" || true
- if mc ls minio/${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET} > /dev/null 2>&1; then
- echo "MinIO bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}' already exists."
- else
- mc mb minio/${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}
- echo "MinIO bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}' created successfully."
- fi
+{{- if .Values.lakekeeperInit.createBucket }}
+ echo "Step 1: Initializing object-store bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}'..."
+ apk add --no-cache wget
+ wget -q https://dl.min.io/client/mc/release/linux-amd64/mc -O /usr/local/bin/mc
+ chmod +x /usr/local/bin/mc
+ mc alias set minio "${STORAGE_S3_ENDPOINT}" "${STORAGE_S3_AUTH_USERNAME}" "${STORAGE_S3_AUTH_PASSWORD}"
+ mc mb --ignore-existing minio/${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}
+ echo "Bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}' is ready."
+{{- else }}
+ echo "Step 1: skipping bucket creation (createBucket=false; using pre-existing external S3 bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}')."
+{{- end }}
echo "Step 2: Initializing default project..."
PROJECT_PAYLOAD="{\"project-id\":\"${LAKEKEEPER_PROJECT_ID}\",\"project-name\":\"${LAKEKEEPER_PROJECT_NAME}\"}"
@@ -112,9 +116,14 @@ spec:
"type": "s3",
"bucket": "${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}",
"region": "${STORAGE_ICEBERG_CATALOG_REST_REGION}",
+{{- if .Values.lakekeeperInit.warehouse.keyPrefix }}
+ "key-prefix": {{ .Values.lakekeeperInit.warehouse.keyPrefix | quote }},
+{{- end }}
+{{- if eq .Values.lakekeeperInit.warehouse.flavor "s3-compat" }}
"endpoint": "${STORAGE_S3_ENDPOINT}",
- "flavor": "s3-compat",
"path-style-access": true,
+{{- end }}
+ "flavor": {{ .Values.lakekeeperInit.warehouse.flavor | quote }},
"sts-enabled": false
},
"storage-credential": {
diff --git a/bin/k8s/templates/on-prem/minio-persistence.yaml b/bin/k8s/templates/on-prem/minio-persistence.yaml
index 94d22e5e3a..4073c32582 100644
--- a/bin/k8s/templates/on-prem/minio-persistence.yaml
+++ b/bin/k8s/templates/on-prem/minio-persistence.yaml
@@ -19,7 +19,7 @@
{{/* This path only works for local-path storage class */}}
{{- $hostBasePath := .Values.persistence.minioHostLocalPath }}
-{{- if .Values.minio.persistence.enabled }}
+{{- if and .Values.minio.enabled .Values.minio.persistence.enabled }}
{{- $name := "minio" }}
{{- $persistence := .Values.minio.persistence }}
{{- $volumeName := printf "%s-data-pv" $name }}
diff --git a/bin/k8s/values-aws.yaml b/bin/k8s/values-aws.yaml
new file mode 100644
index 0000000000..8d1b0a3698
--- /dev/null
+++ b/bin/k8s/values-aws.yaml
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# ---------------------------------------------------------------------------
+# Example overlay for running Texera against an external S3 store (e.g. AWS S3)
+# instead of the bundled in-cluster MinIO.
+#
+# helm install texera bin/k8s -f bin/k8s/values-aws.yaml
+#
+# Replace the placeholder bucket names, region, and credentials below with your
+# own. The buckets must already exist (the init job does not create them).
+# ---------------------------------------------------------------------------
+
+# Turn off the bundled MinIO object store; the services talk to external S3.
+minio:
+ enabled: false
+ persistence:
+ enabled: false
+
+# External S3 object store shared by the app tier, LakeFS, and Lakekeeper.
+storage:
+ s3:
+ # Regional S3 endpoint. For AWS S3 use https://s3.<region>.amazonaws.com.
+ endpoint: "https://s3.us-west-2.amazonaws.com"
+ region: "us-west-2"
+ # Recommended: reference a pre-created Secret with keys access-key-id /
+ # secret-access-key (e.g. one synced from AWS Secrets Manager), and leave
+ # accessKeyId / secretAccessKey empty. Otherwise the chart creates a Secret
+ # named "<release>-s3-credentials" from the inline values below.
+ existingSecret: ""
+ accessKeyId: "REPLACE_WITH_ACCESS_KEY_ID"
+ secretAccessKey: "REPLACE_WITH_SECRET_ACCESS_KEY"
+
+# Point LakeFS at external S3: a region-only blockstore (the AWS SDK resolves
+# the endpoint from the region) with credentials injected via the same Secret
+# the chart/app uses.
+#
+# NOTE: the secretKeyRef "name" below is "texera-s3-credentials", which assumes
+# the Helm release is named "texera" (the chart generates the credentials Secret
+# as "<release>-s3-credentials"). If you install with a different release name,
+# or set storage.s3.existingSecret above, update both "name:" fields below to
+# match that Secret.
+lakefs:
+ lakefsConfig: |
+ database:
+ type: postgres
+ blockstore:
+ type: s3
+ s3:
+ region: us-west-2
+ pre_signed_expiry: 15m
+ extraEnvVars:
+ - name: AWS_ACCESS_KEY_ID
+ valueFrom:
+ secretKeyRef:
+ name: texera-s3-credentials
+ key: access-key-id
+ - name: AWS_SECRET_ACCESS_KEY
+ valueFrom:
+ secretKeyRef:
+ name: texera-s3-credentials
+ key: secret-access-key
+
+# Lakekeeper Iceberg warehouse on external S3: skip bucket creation (the bucket
+# pre-exists and the IAM principal may lack CreateBucket) and use the native
+# AWS S3 storage-profile flavor (no custom endpoint / path-style access).
+lakekeeperInit:
+ createBucket: false
+ warehouse:
+ region: us-west-2
+ s3Bucket: texera-iceberg
+ flavor: aws
diff --git a/bin/k8s/values.yaml b/bin/k8s/values.yaml
index 64642b517a..4651bae694 100644
--- a/bin/k8s/values.yaml
+++ b/bin/k8s/values.yaml
@@ -78,6 +78,10 @@ storage:
secretAccessKey: ""
minio:
+ # Set to false to disable the in-cluster MinIO and point the services at an
+ # external S3 store instead (configure storage.s3 above and the lakefs/
+ # lakekeeperInit blocks below). See values-aws.yaml for a complete example.
+ enabled: true
mode: standalone
image:
repository: bitnamilegacy/minio
@@ -102,6 +106,9 @@ minio:
existingClaim: "minio-data-pvc"
lakefs:
+ # The blockstore below points LakeFS at the in-cluster MinIO. To use an
+ # external S3 store, override lakefsConfig (region-only blockstore) and inject
+ # the S3 credentials via extraEnvVars -- see values-aws.yaml for an example.
secrets:
authEncryptSecretKey: random_string_for_lakefs
databaseConnectionString: postgres://postgres:root_password@texera-postgresql:5432/texera_lakefs?sslmode=disable
@@ -149,6 +156,10 @@ lakekeeper:
lakekeeperInit:
enabled: true
+ # Create the object-store bucket from the init job (true for in-cluster MinIO).
+ # Set false when pointing at a pre-existing external bucket (e.g. AWS S3) whose
+ # IAM principal can't CreateBucket.
+ createBucket: true
defaultProject:
id: "00000000-0000-0000-0000-000000000000"
name: default
@@ -156,6 +167,12 @@ lakekeeperInit:
name: texera
region: us-west-2
s3Bucket: texera-iceberg
+ # Lakekeeper S3 storage-profile flavor: "s3-compat" (MinIO/other; adds
+ # endpoint + path-style-access) or "aws" (real AWS S3; omits both).
+ flavor: s3-compat
+ # Optional object-key prefix to isolate the Iceberg warehouse within a
+ # shared bucket. Empty = bucket root.
+ keyPrefix: ""
# Part2: configurations of Texera-related micro services
texeraImages: