You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by ze...@apache.org on 2026/10/21 21:29:02 UTC
(iceberg-go) branch main updated: test(catalog/rest): run conformance suite against external REST catalogs, with Polaris in CI (#2190)
This is an automated email from the ASF dual-hosted git repository.
zeroshade pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-go.git
The following commit(s) were added to refs/heads/main by this push:
new 7eea50c9b test(catalog/rest): run conformance suite against external REST catalogs, with Polaris in CI (#2190)
7eea50c9b is described below
commit 7eea50c9b7dcbe5bc2644b38a4e9f216b2f66ab9
Author: Richie Bachala <ri...@gmail.com>
AuthorDate: Wed Oct 21 14:28:56 2026 -0700
test(catalog/rest): run conformance suite against external REST catalogs, with Polaris in CI (#2190)
* test(catalog/rest): run conformance suite against Apache Polaris
Add an opt-in integration test that runs the catalog conformance suite
against an Apache Polaris server, a second REST implementation alongside
the Java REST fixture. Testing against two servers surfaces client-side
divergences that a single fixture can mask, such as the rename error
mappings fixed in #2165 and #2173.
The test is skipped unless POLARIS_URI is set, so the integration job is
unaffected. POLARIS_CREDENTIAL and POLARIS_WAREHOUSE are required once
it is enabled. CONTRIBUTING.md documents starting a local Polaris server
with file storage, including the CATALOG_MANAGE_CONTENT grant the client
needs because it requests vended credentials by default.
Signed-off-by: Richie Bachala <ri...@gmail.com>
* test(catalog/rest): move Polaris setup into a script
Address review feedback on the Polaris conformance test:
- Move the bootstrap out of CONTRIBUTING.md into
internal/recipe/setup-polaris.sh, next to the compose file so CI can
reuse it. CONTRIBUTING keeps a short pointer.
- The script bounds the readiness wait and prints the container logs if
Polaris never comes up, fails on a missing token (jq -e) and surfaces
error bodies (curl --fail-with-body), and replaces its container on
each run so it can be rerun.
- Explain the readiness override: Polaris refuses to start with FILE
storage otherwise, since it flags FILE as a severe security risk.
- Require POLARIS_CREDENTIAL to be a client_id:client_secret pair.
- Log which outcome RenameTableToSelf took, so a backend's self-rename
behavior is visible in a passing run.
Signed-off-by: Richie Bachala <ri...@gmail.com>
* test(catalog/rest): show Polaris error bodies and log real self-rename error
- setup-polaris.sh: route API calls through a helper that prints the
server's error body on failure. --fail-with-body alone did not surface
it, because each body went into jq or /dev/null. Use 2-space indents
to match the repository's other shell scripts.
- RenameTableToSelf: log the error a rejected self-rename returned,
rather than a fixed message that would misreport an unexpected error.
Signed-off-by: Richie Bachala <ri...@gmail.com>
* test(catalog/rest): generalize Polaris conformance into an external REST catalog test
Per review on #2190, replace TestPolarisCatalogConformance with
TestExternalRestCatalogConformance, which runs the conformance suite against
any REST catalog configured through REST_CATALOG_URI, _CREDENTIAL, _SCOPE and
_WAREHOUSE. Only the URI is required.
Add make integration-setup-polaris and integration-external-rest, which
default to the local Polaris server, and run them in the Go Integration
workflow on the Spark 3.5 leg, since Polaris does not depend on Spark.
Signed-off-by: Richie Bachala <ri...@gmail.com>
---------
Signed-off-by: Richie Bachala <ri...@gmail.com>
---
.github/workflows/go-integration.yml | 12 ++-
CONTRIBUTING.md | 16 ++++
Makefile | 19 ++++-
catalog/catalogtest/tables.go | 3 +
catalog/rest/external_conformance_test.go | 89 +++++++++++++++++++++++
internal/recipe/setup-polaris.sh | 117 ++++++++++++++++++++++++++++++
6 files changed, 254 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/go-integration.yml b/.github/workflows/go-integration.yml
index bf5e15926..77e4e7958 100644
--- a/.github/workflows/go-integration.yml
+++ b/.github/workflows/go-integration.yml
@@ -83,9 +83,19 @@ jobs:
run: |
make integration-test
+ # Polaris does not depend on Spark, so the external REST catalog
+ # conformance suite runs on one matrix leg only.
+ - name: Run external REST catalog conformance tests (Apache Polaris)
+ if: ${{ matrix.spark.version == '3.5' }}
+ run: |
+ make integration-setup-polaris
+ make integration-external-rest
+
- name: Show debug logs
if: ${{ failure() }}
- run: docker compose -f internal/recipe/docker-compose.yml logs
+ run: |
+ docker compose -f internal/recipe/docker-compose.yml logs
+ docker logs iceberg-go-polaris || true
- name: Save Go module and build cache
if: github.ref == 'refs/heads/main'
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 85efa66f3..9cab72742 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -64,3 +64,19 @@ Integration tests require Docker and are gated behind a build tag:
docker compose -f internal/recipe/docker-compose.yml up -d rest object-store create-bucket --wait
go test -tags integration ./...
```
+
+#### External REST Catalog Conformance
+
+`TestExternalRestCatalogConformance` runs the catalog conformance suite against any REST catalog server. It is skipped unless `REST_CATALOG_URI` is set. CI runs it against [Apache Polaris](https://polaris.apache.org/); to do the same locally (requires `jq`):
+
+```bash
+make integration-setup-polaris
+make integration-external-rest
+```
+
+To test another catalog, override the Polaris defaults with `REST_CATALOG_URI`, `REST_CATALOG_CREDENTIAL` (`client_id:client_secret`), `REST_CATALOG_SCOPE` and `REST_CATALOG_WAREHOUSE`. Set any of them to an empty value to leave it unset, for example:
+
+```bash
+REST_CATALOG_URI=http://localhost:8181 REST_CATALOG_CREDENTIAL= REST_CATALOG_SCOPE= REST_CATALOG_WAREHOUSE= \
+ make integration-external-rest
+```
diff --git a/Makefile b/Makefile
index 162278efb..104f645e0 100644
--- a/Makefile
+++ b/Makefile
@@ -17,7 +17,7 @@
# golangci-lint version (keep in sync with CI and README)
GOLANGCI_LINT_VERSION := v2.14.0
-.PHONY: test test-assert test-race vet lint lint-install integration-setup integration-setup-spark4 integration-test integration-scanner integration-io integration-rest integration-rest-scan-planning integration-spark integration-hadoop integration-down integration-logs docs-gen
+.PHONY: test test-assert test-race vet lint lint-install integration-setup integration-setup-spark4 integration-test integration-scanner integration-io integration-rest integration-rest-scan-planning integration-setup-polaris integration-external-rest integration-spark integration-hadoop integration-down integration-logs docs-gen
test:
go test -v ./...
@@ -83,6 +83,23 @@ integration-rest:
integration-rest-scan-planning:
RUN_INTEGRATION_TESTS=1 go test -tags=integration -v -run="^TestRestIntegration/TestScanPlanningJava" ./catalog/rest
+# integration-external-rest runs the catalog conformance suite against an
+# external REST catalog server. The defaults match the Polaris server that
+# integration-setup-polaris starts. Set REST_CATALOG_* in the environment to
+# point it at another catalog; set one to an empty value to leave it unset.
+REST_CATALOG_URI ?= http://localhost:8191/api/catalog
+REST_CATALOG_CREDENTIAL ?= root:s3cr3t
+REST_CATALOG_SCOPE ?= PRINCIPAL_ROLE:ALL
+REST_CATALOG_WAREHOUSE ?= conformance
+
+integration-setup-polaris:
+ internal/recipe/setup-polaris.sh
+
+integration-external-rest:
+ REST_CATALOG_URI='$(REST_CATALOG_URI)' REST_CATALOG_CREDENTIAL='$(REST_CATALOG_CREDENTIAL)' \
+ REST_CATALOG_SCOPE='$(REST_CATALOG_SCOPE)' REST_CATALOG_WAREHOUSE='$(REST_CATALOG_WAREHOUSE)' \
+ go test -tags=integration -v -run="^TestExternalRestCatalogConformance$$" ./catalog/rest
+
integration-spark:
go test -tags=integration -v -run="^TestSparkIntegration" ./table
diff --git a/catalog/catalogtest/tables.go b/catalog/catalogtest/tables.go
index ded4b6dc7..664357e83 100644
--- a/catalog/catalogtest/tables.go
+++ b/catalog/catalogtest/tables.go
@@ -286,6 +286,9 @@ func testRenameTableToSelf(t *testing.T, cfg Config) {
if err != nil {
assert.ErrorIs(t, err, catalog.ErrTableAlreadyExists,
"a rejected self-rename should report that the table already exists")
+ t.Logf("self-rename rejected: %v", err)
+ } else {
+ t.Log("self-rename succeeded as a no-op")
}
tbl, err := cat.LoadTable(ctx, ident)
diff --git a/catalog/rest/external_conformance_test.go b/catalog/rest/external_conformance_test.go
new file mode 100644
index 000000000..d76e82714
--- /dev/null
+++ b/catalog/rest/external_conformance_test.go
@@ -0,0 +1,89 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//go:build integration
+
+package rest_test
+
+import (
+ "context"
+ "os"
+ "testing"
+
+ "github.com/apache/iceberg-go"
+ "github.com/apache/iceberg-go/catalog"
+ "github.com/apache/iceberg-go/catalog/catalogtest"
+ "github.com/stretchr/testify/require"
+)
+
+// Environment variables that point the conformance suite at an external REST
+// catalog server, alongside the Java REST fixture the suite runs against by
+// default. Only REST_CATALOG_URI is required; the others are passed to the
+// client when set. CI runs this test against Apache Polaris, which
+// internal/recipe/setup-polaris.sh starts; see "make integration-external-rest".
+const (
+ // externalURIEnv is the catalog base URI, including any path prefix the
+ // server needs, for example http://localhost:8191/api/catalog for Polaris.
+ // The OAuth token endpoint is derived from it.
+ externalURIEnv = "REST_CATALOG_URI"
+ // externalCredentialEnv is an OAuth client_id:client_secret pair.
+ externalCredentialEnv = "REST_CATALOG_CREDENTIAL"
+ // externalScopeEnv is the OAuth scope to request. Polaris, for example,
+ // rejects the client's default "catalog" scope and needs PRINCIPAL_ROLE:ALL.
+ externalScopeEnv = "REST_CATALOG_SCOPE"
+ // externalWarehouseEnv selects the warehouse, or the catalog for servers
+ // such as Polaris that host several.
+ externalWarehouseEnv = "REST_CATALOG_WAREHOUSE"
+)
+
+// TestExternalRestCatalogConformance runs the catalog conformance suite against
+// an external REST catalog server. It is opt-in: the test is skipped unless
+// REST_CATALOG_URI is set.
+func TestExternalRestCatalogConformance(t *testing.T) {
+ uri := os.Getenv(externalURIEnv)
+ if uri == "" {
+ t.Skipf("%s is not set; skipping external REST catalog conformance tests", externalURIEnv)
+ }
+
+ props := iceberg.Properties{"type": "rest", "uri": uri}
+ if credential := os.Getenv(externalCredentialEnv); credential != "" {
+ require.Contains(t, credential, ":", "%s must be a client_id:client_secret pair", externalCredentialEnv)
+ props["credential"] = credential
+ }
+ if scope := os.Getenv(externalScopeEnv); scope != "" {
+ props["scope"] = scope
+ }
+ if warehouse := os.Getenv(externalWarehouseEnv); warehouse != "" {
+ props["warehouse"] = warehouse
+ }
+
+ catalogtest.RunCatalogTests(t, catalogtest.Config{
+ SupportsNamespaceProperties: true,
+ SupportsRenameTable: true,
+ NewCatalog: func(t *testing.T) catalog.Catalog {
+ cat, err := catalog.Load(context.Background(), "external", props)
+ require.NoError(t, err)
+ t.Cleanup(func() {
+ if closer, ok := cat.(catalog.Closer); ok {
+ require.NoError(t, closer.Close())
+ }
+ })
+
+ return cat
+ },
+ })
+}
diff --git a/internal/recipe/setup-polaris.sh b/internal/recipe/setup-polaris.sh
new file mode 100755
index 000000000..1da88be3e
--- /dev/null
+++ b/internal/recipe/setup-polaris.sh
@@ -0,0 +1,117 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Starts a local Apache Polaris server for TestExternalRestCatalogConformance
+# and creates a file-backed catalog the conformance suite can use.
+#
+# Usage: internal/recipe/setup-polaris.sh
+#
+# Requires docker, curl and jq. Each run replaces the container, so the script
+# can be rerun safely. On success it prints the environment variables the test
+# reads. The credentials are for local testing only.
+
+set -euo pipefail
+
+image=apache/polaris:1.8.0
+container=iceberg-go-polaris
+api_port=8191
+mgmt_port=8192
+client_id=root
+client_secret=s3cr3t
+warehouse=conformance
+location=file:///tmp/polaris/conformance
+
+for cmd in docker curl jq; do
+ command -v "$cmd" > /dev/null || {
+ echo "setup-polaris: $cmd is required" >&2
+ exit 1
+ }
+done
+
+# call runs curl and prints the response body. On an HTTP error it reports the
+# server's error body, which would otherwise be lost in a pipe or /dev/null.
+call() {
+ local body
+ if ! body=$(curl -sS --fail-with-body "$@"); then
+ echo "setup-polaris: request failed: ${body:-no response body}" >&2
+ exit 1
+ fi
+ printf '%s' "$body"
+}
+
+docker rm -f "$container" > /dev/null 2>&1 || true
+
+# Polaris refuses to start with FILE storage unless the readiness check is
+# told to ignore it: FILE storage is flagged as a severe security risk, which
+# is acceptable for a throwaway local server.
+docker run -d --name "$container" \
+ -p "$api_port:8181" -p "$mgmt_port:8182" \
+ -e POLARIS_BOOTSTRAP_CREDENTIALS="POLARIS,$client_id,$client_secret" \
+ -e 'polaris.features."ALLOW_INSECURE_STORAGE_TYPES"=true' \
+ -e 'polaris.features."SUPPORTED_CATALOG_STORAGE_TYPES"=["FILE"]' \
+ -e 'polaris.readiness.ignore-severe-issues=true' \
+ "$image" > /dev/null
+
+ready=false
+for _ in $(seq 60); do
+ if curl -sf "localhost:$mgmt_port/q/health/ready" > /dev/null; then
+ ready=true
+ break
+ fi
+ sleep 1
+done
+if [[ $ready != true ]]; then
+ echo "setup-polaris: Polaris did not become ready within 60s; recent logs:" >&2
+ docker logs --tail 20 "$container" >&2 || true
+ exit 1
+fi
+
+api="http://localhost:$api_port/api"
+
+# jq -e fails when the response has no access_token instead of passing an empty
+# or "null" token to the calls below.
+token=$(call -X POST "$api/catalog/v1/oauth/tokens" \
+ -d grant_type=client_credentials -d client_id="$client_id" \
+ -d client_secret="$client_secret" -d scope=PRINCIPAL_ROLE:ALL |
+ jq -er .access_token)
+
+call -X POST "$api/management/v1/catalogs" \
+ -H "Authorization: Bearer $token" -H 'Content-Type: application/json' \
+ -d "{\"catalog\":{\"name\":\"$warehouse\",\"type\":\"INTERNAL\",
+ \"properties\":{\"default-base-location\":\"$location\"},
+ \"storageConfigInfo\":{\"storageType\":\"FILE\",\"allowedLocations\":[\"$location\"]}}}" \
+ > /dev/null
+
+# The REST client requests vended credentials (X-Iceberg-Access-Delegation) by
+# default, which Polaris authorizes with TABLE_WRITE_DATA. CATALOG_MANAGE_CONTENT
+# includes it.
+call -X PUT "$api/management/v1/catalogs/$warehouse/catalog-roles/catalog_admin/grants" \
+ -H "Authorization: Bearer $token" -H 'Content-Type: application/json' \
+ -d '{"grant":{"type":"catalog","privilege":"CATALOG_MANAGE_CONTENT"}}' \
+ > /dev/null
+
+cat << EOF
+Polaris is running in container $container. Run the conformance suite with:
+
+ REST_CATALOG_URI=$api/catalog \\
+ REST_CATALOG_CREDENTIAL=$client_id:$client_secret \\
+ REST_CATALOG_SCOPE=PRINCIPAL_ROLE:ALL \\
+ REST_CATALOG_WAREHOUSE=$warehouse \\
+ go test -tags integration -run TestExternalRestCatalogConformance ./catalog/rest
+
+Remove the server with: docker rm -f $container
+EOF