You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/07/14 01:02:45 UTC
(texera) branch main updated: feat(k8s): external S3 for LakeFS & Lakekeeper, MinIO optional (#6295)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new 8770ab87ba feat(k8s): external S3 for LakeFS & Lakekeeper, MinIO optional (#6295)
8770ab87ba is described below
commit 8770ab87bade26e452b87a3979e4c78c0ea0bdd5
Author: ali risheh <al...@gmail.com>
AuthorDate: Mon Jul 13 17:52:32 2026 -0700
feat(k8s): external S3 for LakeFS & Lakekeeper, MinIO optional (#6295)
### What changes were proposed in this PR?
This completes the "make object storage pluggable" work by wiring the
**data tier** (LakeFS blockstore + Lakekeeper Iceberg warehouse) to an
external S3 store and adding a single switch to drop the in-cluster
MinIO entirely. #5932 did the **application tier** (`storage.s3`) and
explicitly deferred this half. It is **non-breaking**: the default
(on-prem / in-cluster MinIO) install renders the same set of resources.
- **`minio.enabled` switch** (`values.yaml`, default `true`) to turn off
the bundled MinIO subchart and point everything at external S3:
- `Chart.yaml` -- the `minio` dependency gets `condition:
minio.enabled`.
- `templates/on-prem/minio-persistence.yaml` -- the MinIO PV/PVC is
gated on `minio.enabled` as well as `minio.persistence.enabled`.
- `templates/base/external-names/external-names.yaml` -- the
`<release>-minio` `ExternalName` alias (exposed to the
computing-unit-pool namespace) is only rendered when MinIO is enabled;
with external S3 the CU pods reach the store directly via
`STORAGE_S3_ENDPOINT`.
- **Lakekeeper init job**
(`templates/base/lakekeeper/lakekeeper-init-job.yaml`):
- `STORAGE_S3_*` env now resolves through the shared `texera.s3.*`
helpers introduced in #5932 (endpoint + credentials Secret name/keys)
instead of hardcoding `<release>-minio`.
- New `lakekeeperInit.createBucket` (default `true`) -- when `false`,
the job skips `mc mb` and logs that it is using a pre-existing external
bucket. Needed because an external bucket already exists and its IAM
principal may lack `s3:CreateBucket`.
- New `lakekeeperInit.warehouse.flavor` -- `s3-compat` (MinIO/other;
emits `endpoint` + `path-style-access: true`) or native `aws` (real AWS
S3; omits both). New optional `lakekeeperInit.warehouse.keyPrefix` to
isolate the warehouse within a shared bucket.
- **`values.yaml`** -- documents `minio.enabled`, the LakeFS
external-blockstore override path, and the new `lakekeeperInit` keys.
- **`values-aws.yaml`** (new) -- a complete, copy-pasteable
all-external-S3 overlay: MinIO off, `storage.s3`
(endpoint/region/credentials or `existingSecret`), LakeFS region-only S3
blockstore with credentials injected via `extraEnvVars`, and Lakekeeper
`flavor: aws` + `createBucket: false`.
How it behaves:
- **Default (no config):** LakeFS and Lakekeeper use the in-cluster
MinIO exactly as before; the MinIO subchart, PV/PVC, and `ExternalName`
all render as today.
- **External S3 (`-f values-aws.yaml`):** no MinIO is deployed; the app
tier, LakeFS, and Lakekeeper all target the external S3 buckets, and the
Lakekeeper warehouse is registered with the native AWS flavor against a
pre-existing bucket.
### Any related issues, documentation, discussions?
Closes #6294 (LakeFS blockstore + Lakekeeper warehouse external-S3
task).
Part of #5891 -- unify AWS (EKS) and on-premise Kubernetes deployment
under `bin/k8s` (parent feature).
Follows #5932 (app-tier `storage.s3`) and #5757 (Helm template reorg).
### How was this PR tested?
**1. Render parity (default install unchanged).** `helm template texera
bin/k8s` on this branch vs on `main` renders the same set of resources;
`helm lint bin/k8s` passes. The new `{{- if }}` guards (MinIO
ExternalName, persistence, createBucket, flavor) all evaluate to their
prior behavior when `minio.enabled`/`createBucket` default to `true`.
**2. External-S3 render.** `helm template texera bin/k8s -f
bin/k8s/values-aws.yaml` deploys no MinIO subchart / PV / PVC /
ExternalName, routes the Lakekeeper init job's `STORAGE_S3_*` at the
external endpoint and credentials Secret, skips the `mc mb`
bucket-creation step, and registers the warehouse storage profile with
`"flavor": "aws"` (no `endpoint` / `path-style-access`). `helm lint`
passes for this value set.
**3. End-to-end on a live EKS cluster with real AWS S3.** Stood up an
isolated EKS cluster (`us-west-2`, its own AWS account, no MinIO) plus
two pre-created S3 buckets (LakeFS blockstore + Iceberg warehouse) and
an IAM user scoped to just those buckets with **no** `CreateBucket`
permission. Built all Texera service images from this branch, deployed
the chart with an external-S3 overlay derived from `values-aws.yaml`,
and exercised the full stack:
- LakeFS + Lakekeeper init jobs authenticated to external S3 and
initialized (Lakekeeper `aws` flavor, bucket-creation skipped against
the pre-existing warehouse bucket).
- Created a dataset -> objects written to the external LakeFS bucket;
dataset preview served via presigned S3 URLs (external buckets need a
CORS rule, noted below).
- Ran a workflow (CSV file scan) end to end -> **success, 100 rows
returned**, with the Iceberg result table physically materialized in the
external warehouse bucket (`metadata.json` + data objects confirmed via
the S3 API). This exercises the full external-S3 path: file-service
reads the CSV from LakeFS -> S3, the computing-unit pod reaches the
Lakekeeper REST catalog, and results are written to Iceberg -> S3 and
read back.
All test AWS resources (cluster, buckets, IAM user) were torn down
afterward.
Operational note surfaced by the live test (not a chart defect, worth
documenting): with a real external S3 store, dataset preview requires a
CORS rule on the buckets (`GET`/`HEAD`) because the presigned URLs are
served cross-origin from `*.s3.<region>.amazonaws.com`, whereas the
in-cluster MinIO was same-origin.
No unit tests were added -- the change is limited to Helm chart
values/templates, validated by the render diff, `helm lint`, and the
live end-to-end run above.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
---------
Co-authored-by: Claude Opus 4.8 (1M context) <no...@anthropic.com>
---
bin/k8s/Chart.yaml | 1 +
.../base/external-names/external-names.yaml | 7 +-
.../base/lakekeeper/lakekeeper-init-job.yaml | 43 ++++++-----
bin/k8s/templates/on-prem/minio-persistence.yaml | 2 +-
bin/k8s/values-aws.yaml | 86 ++++++++++++++++++++++
bin/k8s/values.yaml | 17 +++++
6 files changed, 136 insertions(+), 20 deletions(-)
diff --git a/bin/k8s/Chart.yaml b/bin/k8s/Chart.yaml
index 548d9e531e..8cda7ee227 100644
--- a/bin/k8s/Chart.yaml
+++ b/bin/k8s/Chart.yaml
@@ -49,6 +49,7 @@ dependencies:
- name: minio
version: 15.0.7
repository: https://charts.bitnami.com/bitnami
+ condition: minio.enabled
- name: lakefs
version: 1.8.1
diff --git a/bin/k8s/templates/base/external-names/external-names.yaml b/bin/k8s/templates/base/external-names/external-names.yaml
index c79ce40f33..6879b8d7f7 100644
--- a/bin/k8s/templates/base/external-names/external-names.yaml
+++ b/bin/k8s/templates/base/external-names/external-names.yaml
@@ -73,13 +73,16 @@ to access services in the main namespace using the same service names.
"externalName" (printf "%s-svc.%s.svc.cluster.local" .Values.webserver.name $namespace)
) | nindent 0 }}
+{{- if .Values.minio.enabled }}
---
-{{/* MinIO ExternalName */}}
-{{- include "external-name-service" (dict
+{{/* MinIO ExternalName -- only when the in-cluster MinIO is enabled; with an
+ external S3 store the CU pods reach it directly via STORAGE_S3_ENDPOINT. */}}
+{{- include "external-name-service" (dict
"name" (printf "%s-minio" .Release.Name)
"namespace" $workflowComputingUnitPoolNamespace
"externalName" (printf "%s-minio.%s.svc.cluster.local" .Release.Name $namespace)
) | nindent 0 }}
+{{- end }}
---
{{/* Lakekeeper ExternalName */}}
diff --git a/bin/k8s/templates/base/lakekeeper/lakekeeper-init-job.yaml b/bin/k8s/templates/base/lakekeeper/lakekeeper-init-job.yaml
index 0a3540b605..d1adafcd92 100644
--- a/bin/k8s/templates/base/lakekeeper/lakekeeper-init-job.yaml
+++ b/bin/k8s/templates/base/lakekeeper/lakekeeper-init-job.yaml
@@ -16,6 +16,9 @@
# under the License.
{{- if .Values.lakekeeperInit.enabled }}
+{{- if not (has .Values.lakekeeperInit.warehouse.flavor (list "s3-compat" "aws")) }}
+{{- fail (printf "lakekeeperInit.warehouse.flavor must be \"s3-compat\" or \"aws\", got %q" .Values.lakekeeperInit.warehouse.flavor) }}
+{{- end }}
apiVersion: batch/v1
kind: Job
metadata:
@@ -33,17 +36,17 @@ spec:
image: alpine:3.19
env:
- name: STORAGE_S3_ENDPOINT
- value: http://{{ .Release.Name }}-minio:9000
+ value: {{ include "texera.s3.endpoint" . }}
- name: STORAGE_S3_AUTH_USERNAME
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-user
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.accessKeyIdKey" . }}
- name: STORAGE_S3_AUTH_PASSWORD
valueFrom:
secretKeyRef:
- name: {{ .Release.Name }}-minio
- key: root-password
+ name: {{ include "texera.s3.secretName" . }}
+ key: {{ include "texera.s3.secretAccessKeyKey" . }}
- name: STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET
value: {{ .Values.lakekeeperInit.warehouse.s3Bucket | quote }}
- name: STORAGE_ICEBERG_CATALOG_REST_REGION
@@ -62,9 +65,7 @@ spec:
- |
set -e
- apk add --no-cache curl ca-certificates wget
- wget -q https://dl.min.io/client/mc/release/linux-amd64/mc -O /usr/local/bin/mc
- chmod +x /usr/local/bin/mc
+ apk add --no-cache curl ca-certificates
check_status() {
if [ "$1" -ge 200 ] && [ "$1" -lt 300 ]; then
@@ -85,14 +86,17 @@ spec:
sleep 3
done
- echo "Step 1: Initializing MinIO bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}'..."
- mc alias set minio "${STORAGE_S3_ENDPOINT}" "${STORAGE_S3_AUTH_USERNAME}" "${STORAGE_S3_AUTH_PASSWORD}" || true
- if mc ls minio/${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET} > /dev/null 2>&1; then
- echo "MinIO bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}' already exists."
- else
- mc mb minio/${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}
- echo "MinIO bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}' created successfully."
- fi
+{{- if .Values.lakekeeperInit.createBucket }}
+ echo "Step 1: Initializing object-store bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}'..."
+ apk add --no-cache wget
+ wget -q https://dl.min.io/client/mc/release/linux-amd64/mc -O /usr/local/bin/mc
+ chmod +x /usr/local/bin/mc
+ mc alias set minio "${STORAGE_S3_ENDPOINT}" "${STORAGE_S3_AUTH_USERNAME}" "${STORAGE_S3_AUTH_PASSWORD}"
+ mc mb --ignore-existing minio/${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}
+ echo "Bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}' is ready."
+{{- else }}
+ echo "Step 1: skipping bucket creation (createBucket=false; using pre-existing external S3 bucket '${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}')."
+{{- end }}
echo "Step 2: Initializing default project..."
PROJECT_PAYLOAD="{\"project-id\":\"${LAKEKEEPER_PROJECT_ID}\",\"project-name\":\"${LAKEKEEPER_PROJECT_NAME}\"}"
@@ -112,9 +116,14 @@ spec:
"type": "s3",
"bucket": "${STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET}",
"region": "${STORAGE_ICEBERG_CATALOG_REST_REGION}",
+{{- if .Values.lakekeeperInit.warehouse.keyPrefix }}
+ "key-prefix": {{ .Values.lakekeeperInit.warehouse.keyPrefix | quote }},
+{{- end }}
+{{- if eq .Values.lakekeeperInit.warehouse.flavor "s3-compat" }}
"endpoint": "${STORAGE_S3_ENDPOINT}",
- "flavor": "s3-compat",
"path-style-access": true,
+{{- end }}
+ "flavor": {{ .Values.lakekeeperInit.warehouse.flavor | quote }},
"sts-enabled": false
},
"storage-credential": {
diff --git a/bin/k8s/templates/on-prem/minio-persistence.yaml b/bin/k8s/templates/on-prem/minio-persistence.yaml
index 94d22e5e3a..4073c32582 100644
--- a/bin/k8s/templates/on-prem/minio-persistence.yaml
+++ b/bin/k8s/templates/on-prem/minio-persistence.yaml
@@ -19,7 +19,7 @@
{{/* This path only works for local-path storage class */}}
{{- $hostBasePath := .Values.persistence.minioHostLocalPath }}
-{{- if .Values.minio.persistence.enabled }}
+{{- if and .Values.minio.enabled .Values.minio.persistence.enabled }}
{{- $name := "minio" }}
{{- $persistence := .Values.minio.persistence }}
{{- $volumeName := printf "%s-data-pv" $name }}
diff --git a/bin/k8s/values-aws.yaml b/bin/k8s/values-aws.yaml
new file mode 100644
index 0000000000..8d1b0a3698
--- /dev/null
+++ b/bin/k8s/values-aws.yaml
@@ -0,0 +1,86 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# ---------------------------------------------------------------------------
+# Example overlay for running Texera against an external S3 store (e.g. AWS S3)
+# instead of the bundled in-cluster MinIO.
+#
+# helm install texera bin/k8s -f bin/k8s/values-aws.yaml
+#
+# Replace the placeholder bucket names, region, and credentials below with your
+# own. The buckets must already exist (the init job does not create them).
+# ---------------------------------------------------------------------------
+
+# Turn off the bundled MinIO object store; the services talk to external S3.
+minio:
+ enabled: false
+ persistence:
+ enabled: false
+
+# External S3 object store shared by the app tier, LakeFS, and Lakekeeper.
+storage:
+ s3:
+ # Regional S3 endpoint. For AWS S3 use https://s3.<region>.amazonaws.com.
+ endpoint: "https://s3.us-west-2.amazonaws.com"
+ region: "us-west-2"
+ # Recommended: reference a pre-created Secret with keys access-key-id /
+ # secret-access-key (e.g. one synced from AWS Secrets Manager), and leave
+ # accessKeyId / secretAccessKey empty. Otherwise the chart creates a Secret
+ # named "<release>-s3-credentials" from the inline values below.
+ existingSecret: ""
+ accessKeyId: "REPLACE_WITH_ACCESS_KEY_ID"
+ secretAccessKey: "REPLACE_WITH_SECRET_ACCESS_KEY"
+
+# Point LakeFS at external S3: a region-only blockstore (the AWS SDK resolves
+# the endpoint from the region) with credentials injected via the same Secret
+# the chart/app uses.
+#
+# NOTE: the secretKeyRef "name" below is "texera-s3-credentials", which assumes
+# the Helm release is named "texera" (the chart generates the credentials Secret
+# as "<release>-s3-credentials"). If you install with a different release name,
+# or set storage.s3.existingSecret above, update both "name:" fields below to
+# match that Secret.
+lakefs:
+ lakefsConfig: |
+ database:
+ type: postgres
+ blockstore:
+ type: s3
+ s3:
+ region: us-west-2
+ pre_signed_expiry: 15m
+ extraEnvVars:
+ - name: AWS_ACCESS_KEY_ID
+ valueFrom:
+ secretKeyRef:
+ name: texera-s3-credentials
+ key: access-key-id
+ - name: AWS_SECRET_ACCESS_KEY
+ valueFrom:
+ secretKeyRef:
+ name: texera-s3-credentials
+ key: secret-access-key
+
+# Lakekeeper Iceberg warehouse on external S3: skip bucket creation (the bucket
+# pre-exists and the IAM principal may lack CreateBucket) and use the native
+# AWS S3 storage-profile flavor (no custom endpoint / path-style access).
+lakekeeperInit:
+ createBucket: false
+ warehouse:
+ region: us-west-2
+ s3Bucket: texera-iceberg
+ flavor: aws
diff --git a/bin/k8s/values.yaml b/bin/k8s/values.yaml
index 64642b517a..4651bae694 100644
--- a/bin/k8s/values.yaml
+++ b/bin/k8s/values.yaml
@@ -78,6 +78,10 @@ storage:
secretAccessKey: ""
minio:
+ # Set to false to disable the in-cluster MinIO and point the services at an
+ # external S3 store instead (configure storage.s3 above and the lakefs/
+ # lakekeeperInit blocks below). See values-aws.yaml for a complete example.
+ enabled: true
mode: standalone
image:
repository: bitnamilegacy/minio
@@ -102,6 +106,9 @@ minio:
existingClaim: "minio-data-pvc"
lakefs:
+ # The blockstore below points LakeFS at the in-cluster MinIO. To use an
+ # external S3 store, override lakefsConfig (region-only blockstore) and inject
+ # the S3 credentials via extraEnvVars -- see values-aws.yaml for an example.
secrets:
authEncryptSecretKey: random_string_for_lakefs
databaseConnectionString: postgres://postgres:root_password@texera-postgresql:5432/texera_lakefs?sslmode=disable
@@ -149,6 +156,10 @@ lakekeeper:
lakekeeperInit:
enabled: true
+ # Create the object-store bucket from the init job (true for in-cluster MinIO).
+ # Set false when pointing at a pre-existing external bucket (e.g. AWS S3) whose
+ # IAM principal can't CreateBucket.
+ createBucket: true
defaultProject:
id: "00000000-0000-0000-0000-000000000000"
name: default
@@ -156,6 +167,12 @@ lakekeeperInit:
name: texera
region: us-west-2
s3Bucket: texera-iceberg
+ # Lakekeeper S3 storage-profile flavor: "s3-compat" (MinIO/other; adds
+ # endpoint + path-style-access) or "aws" (real AWS S3; omits both).
+ flavor: s3-compat
+ # Optional object-key prefix to isolate the Iceberg warehouse within a
+ # shared bucket. Empty = bucket root.
+ keyPrefix: ""
# Part2: configurations of Texera-related micro services
texeraImages: