You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/07/05 20:08:44 UTC
(texera) 01/01: fix(local-dev): refresh Lakekeeper warehouse S3 endpoint when it already exists (#6197)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6197-26fc1af13e191e1ecf1864f736ea907992e3920f
in repository https://gitbox.apache.org/repos/asf/texera.git
commit 8eb87fb45a3a2b54752b0a71d2c4d54d14b1a823
Author: Jiadong Bai <43...@users.noreply.github.com>
AuthorDate: Sun Jul 5 13:08:23 2026 -0700
fix(local-dev): refresh Lakekeeper warehouse S3 endpoint when it already exists (#6197)
### What changes were proposed in this PR?
Makes `lakekeeper-init` (in `bin/single-node/docker-compose.yml`) keep
the Iceberg warehouse's S3 endpoint in sync with the current
`STORAGE_S3_ENDPOINT`. The warehouse persists its endpoint in
Lakekeeper's own DB; `local-dev` sets `STORAGE_S3_ENDPOINT` to the host
**LAN IP** (not `localhost`) so both the containerized Lakekeeper and
the host JVMs — which reach MinIO directly via Iceberg remote-signing —
use the same address. But that IP changes across networks / DHCP leases,
and the init was idempotent: it **skipped** the warehouse when it
already existed, so the stale endpoint was never refreshed. Workflow
execution then failed with an opaque
`org.apache.iceberg.exceptions.RESTException: Unable to process` at
`createTable`.
Instead of skipping, the existing-warehouse branch now `POST`s the
current endpoint + credentials to
`/management/v1/warehouse/{id}/storage` on every run. This is
non-destructive and preserves existing Iceberg tables (their metadata
stores `s3://bucket/...` paths, not the endpoint).
### Any related issues, documentation, discussions?
Closes #6195
### How was this PR tested?
- Verified the same `POST /management/v1/warehouse/{id}/storage` call
heals a live stack whose warehouse had a stale IP: it returned HTTP 200
and the vended endpoint updated to the current host IP, after which
workflow execution succeeded.
- `docker compose -f bin/single-node/docker-compose.yml config -q`
validates the file.
- Extracted the rendered `lakekeeper-init` command and confirmed `bash
-n` passes (after applying compose's runtime `$$`→`$` unescaping), so
the added heredoc/shell is syntactically sound.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8 (1M context)
---
bin/single-node/docker-compose.yml | 45 +++++++++++++++++++++++++++++++++++++-
1 file changed, 44 insertions(+), 1 deletion(-)
diff --git a/bin/single-node/docker-compose.yml b/bin/single-node/docker-compose.yml
index a7efb81382..64247d9d66 100644
--- a/bin/single-node/docker-compose.yml
+++ b/bin/single-node/docker-compose.yml
@@ -219,7 +219,50 @@ services:
fi
if jq -e --arg name "$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME" '.warehouses[]? | select(.name == $$name)' /tmp/warehouses.txt >/dev/null; then
- echo "Lakekeeper Warehouse '$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME' already exists. Skipping creation."
+ # The warehouse persists its S3 endpoint in Lakekeeper's own DB. The
+ # local-dev launcher sets STORAGE_S3_ENDPOINT to the host LAN IP so
+ # both this container and the host JVMs (Iceberg remote-signing) reach
+ # the same MinIO — but that IP changes across networks / DHCP leases.
+ # This init is idempotent and would otherwise just skip, leaving a
+ # stale endpoint that breaks workflow execution with an opaque Iceberg
+ # RESTException. Refresh the stored endpoint on every run (#6195).
+ echo "Lakekeeper Warehouse '$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME' already exists. Refreshing its storage endpoint to '$$STORAGE_S3_ENDPOINT'..."
+ WAREHOUSE_ID=$$(jq -r --arg name "$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME" '.warehouses[]? | select(.name == $$name) | .id' /tmp/warehouses.txt)
+ UPDATE_PAYLOAD=$$(cat <<EOF
+ {
+ "storage-profile": {
+ "type": "s3",
+ "bucket": "$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET",
+ "region": "$$STORAGE_S3_REGION",
+ "endpoint": "$$STORAGE_S3_ENDPOINT",
+ "flavor": "s3-compat",
+ "path-style-access": true,
+ "sts-enabled": false
+ },
+ "storage-credential": {
+ "type": "s3",
+ "credential-type": "access-key",
+ "aws-access-key-id": "$$STORAGE_S3_AUTH_USERNAME",
+ "aws-secret-access-key": "$$STORAGE_S3_AUTH_PASSWORD"
+ }
+ }
+ EOF
+ )
+
+ UPDATE_CODE=$$(curl -s -o /tmp/response.txt -w "%{http_code}" \
+ -X POST \
+ -H "Content-Type: application/json" \
+ -d "$$UPDATE_PAYLOAD" \
+ "$$LAKEKEEPER_BASE_URI/management/v1/warehouse/$$WAREHOUSE_ID/storage" || echo "000")
+
+ if [ "$$UPDATE_CODE" -lt 200 ] || [ "$$UPDATE_CODE" -ge 300 ]; then
+ echo "Failed to refresh Lakekeeper Warehouse storage. HTTP Code: $$UPDATE_CODE"
+ echo "ERROR RESPONSE:"
+ if [ -f /tmp/response.txt ]; then cat /tmp/response.txt; fi
+ echo ""
+ exit 1
+ fi
+ echo "Refreshed Lakekeeper Warehouse storage endpoint successfully (HTTP $$UPDATE_CODE)."
else
echo "Warehouse not found. Creating '$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME'..."
CREATE_PAYLOAD=$$(cat <<EOF