You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@texera.apache.org by gi...@apache.org on 2026/07/05 20:08:43 UTC
(texera) branch gh-readonly-queue/main/pr-6197-26fc1af13e191e1ecf1864f736ea907992e3920f created (now 8eb87fb45a)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-6197-26fc1af13e191e1ecf1864f736ea907992e3920f
in repository https://gitbox.apache.org/repos/asf/texera.git
at 8eb87fb45a fix(local-dev): refresh Lakekeeper warehouse S3 endpoint when it already exists (#6197)
This branch includes the following new commits:
new 8eb87fb45a fix(local-dev): refresh Lakekeeper warehouse S3 endpoint when it already exists (#6197)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(texera) 01/01: fix(local-dev): refresh Lakekeeper warehouse S3 endpoint when it already exists (#6197)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6197-26fc1af13e191e1ecf1864f736ea907992e3920f
in repository https://gitbox.apache.org/repos/asf/texera.git
commit 8eb87fb45a3a2b54752b0a71d2c4d54d14b1a823
Author: Jiadong Bai <43...@users.noreply.github.com>
AuthorDate: Sun Jul 5 13:08:23 2026 -0700
fix(local-dev): refresh Lakekeeper warehouse S3 endpoint when it already exists (#6197)
### What changes were proposed in this PR?
Makes `lakekeeper-init` (in `bin/single-node/docker-compose.yml`) keep
the Iceberg warehouse's S3 endpoint in sync with the current
`STORAGE_S3_ENDPOINT`. The warehouse persists its endpoint in
Lakekeeper's own DB; `local-dev` sets `STORAGE_S3_ENDPOINT` to the host
**LAN IP** (not `localhost`) so both the containerized Lakekeeper and
the host JVMs — which reach MinIO directly via Iceberg remote-signing —
use the same address. But that IP changes across networks / DHCP leases,
and the init was idempotent: it **skipped** the warehouse when it
already existed, so the stale endpoint was never refreshed. Workflow
execution then failed with an opaque
`org.apache.iceberg.exceptions.RESTException: Unable to process` at
`createTable`.
Instead of skipping, the existing-warehouse branch now `POST`s the
current endpoint + credentials to
`/management/v1/warehouse/{id}/storage` on every run. This is
non-destructive and preserves existing Iceberg tables (their metadata
stores `s3://bucket/...` paths, not the endpoint).
### Any related issues, documentation, discussions?
Closes #6195
### How was this PR tested?
- Verified the same `POST /management/v1/warehouse/{id}/storage` call
heals a live stack whose warehouse had a stale IP: it returned HTTP 200
and the vended endpoint updated to the current host IP, after which
workflow execution succeeded.
- `docker compose -f bin/single-node/docker-compose.yml config -q`
validates the file.
- Extracted the rendered `lakekeeper-init` command and confirmed `bash
-n` passes (after applying compose's runtime `$$`→`$` unescaping), so
the added heredoc/shell is syntactically sound.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8 (1M context)
---
bin/single-node/docker-compose.yml | 45 +++++++++++++++++++++++++++++++++++++-
1 file changed, 44 insertions(+), 1 deletion(-)
diff --git a/bin/single-node/docker-compose.yml b/bin/single-node/docker-compose.yml
index a7efb81382..64247d9d66 100644
--- a/bin/single-node/docker-compose.yml
+++ b/bin/single-node/docker-compose.yml
@@ -219,7 +219,50 @@ services:
fi
if jq -e --arg name "$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME" '.warehouses[]? | select(.name == $$name)' /tmp/warehouses.txt >/dev/null; then
- echo "Lakekeeper Warehouse '$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME' already exists. Skipping creation."
+ # The warehouse persists its S3 endpoint in Lakekeeper's own DB. The
+ # local-dev launcher sets STORAGE_S3_ENDPOINT to the host LAN IP so
+ # both this container and the host JVMs (Iceberg remote-signing) reach
+ # the same MinIO — but that IP changes across networks / DHCP leases.
+ # This init is idempotent and would otherwise just skip, leaving a
+ # stale endpoint that breaks workflow execution with an opaque Iceberg
+ # RESTException. Refresh the stored endpoint on every run (#6195).
+ echo "Lakekeeper Warehouse '$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME' already exists. Refreshing its storage endpoint to '$$STORAGE_S3_ENDPOINT'..."
+ WAREHOUSE_ID=$$(jq -r --arg name "$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME" '.warehouses[]? | select(.name == $$name) | .id' /tmp/warehouses.txt)
+ UPDATE_PAYLOAD=$$(cat <<EOF
+ {
+ "storage-profile": {
+ "type": "s3",
+ "bucket": "$$STORAGE_ICEBERG_CATALOG_REST_S3_BUCKET",
+ "region": "$$STORAGE_S3_REGION",
+ "endpoint": "$$STORAGE_S3_ENDPOINT",
+ "flavor": "s3-compat",
+ "path-style-access": true,
+ "sts-enabled": false
+ },
+ "storage-credential": {
+ "type": "s3",
+ "credential-type": "access-key",
+ "aws-access-key-id": "$$STORAGE_S3_AUTH_USERNAME",
+ "aws-secret-access-key": "$$STORAGE_S3_AUTH_PASSWORD"
+ }
+ }
+ EOF
+ )
+
+ UPDATE_CODE=$$(curl -s -o /tmp/response.txt -w "%{http_code}" \
+ -X POST \
+ -H "Content-Type: application/json" \
+ -d "$$UPDATE_PAYLOAD" \
+ "$$LAKEKEEPER_BASE_URI/management/v1/warehouse/$$WAREHOUSE_ID/storage" || echo "000")
+
+ if [ "$$UPDATE_CODE" -lt 200 ] || [ "$$UPDATE_CODE" -ge 300 ]; then
+ echo "Failed to refresh Lakekeeper Warehouse storage. HTTP Code: $$UPDATE_CODE"
+ echo "ERROR RESPONSE:"
+ if [ -f /tmp/response.txt ]; then cat /tmp/response.txt; fi
+ echo ""
+ exit 1
+ fi
+ echo "Refreshed Lakekeeper Warehouse storage endpoint successfully (HTTP $$UPDATE_CODE)."
else
echo "Warehouse not found. Creating '$$STORAGE_ICEBERG_CATALOG_REST_WAREHOUSE_NAME'..."
CREATE_PAYLOAD=$$(cat <<EOF