You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/05/05 15:32:55 UTC
[PR] Open-API: Slim down REST fixture shadow JAR and externalize cloud bundles [iceberg]
kevinjqliu opened a new pull request, #16214:
URL: https://github.com/apache/iceberg/pull/16214
## Changes
- **Reduce shadow JAR from 201MB to 58MB** by excluding cloud-specific thin modules
(iceberg-aws, iceberg-gcp, iceberg-azure, iceberg-bigquery) and JUnit dependencies
from the shadow JAR.
- **Load cloud bundles as separate JARs** via classpath (`-cp` instead of `-jar`) so
they can be included/excluded independently.
- **Fix `REST_PORT` env var** — renamed to `CATALOG_REST_PORT` so the server actually
reads it through the `CATALOG_*` environment mapping.
- **Update CI workflow** to build cloud bundle shadow JARs alongside the open-api JAR.
## Motivation
The shadow JAR previously bundled all cloud provider SDKs (AWS, GCP, Azure) directly,
making it unnecessarily large. Cloud bundles are already published as standalone JARs and
can be loaded at runtime via the classpath, matching how they're consumed in Spark/Flink.
## Testing
- Built shadow JAR and verified size reduction (201MB → 58MB)
- Ran standalone JAR with `CATALOG_REST_PORT=8183` — server starts, serves `/v1/config`,
creates/lists namespaces
- Built and ran Docker image — healthcheck passes, full REST API functional
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stop publishing REST fixture runtime jar [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#issuecomment-4834756305
After this, I will follow up with more PRs to
1. Prune as many unnecessary dependencies as possible from the rest fixture image
2. Fix the LICENSE/NOTICE file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stop publishing REST fixture runtime jar [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#issuecomment-5210812912
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stage REST fixture runtime JARs for Docker [iceberg]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on code in PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#discussion_r4541988837
##########
.github/workflows/cve-scan.yml:
##########
@@ -194,7 +195,7 @@ jobs:
if [ "${{ matrix.unpack }}" = "true" ]; then
unzip ${{ matrix.scan-path }}/*.zip -d /tmp/cve-scan
else
- cp ${{ matrix.scan-path }}/iceberg-${{ matrix.distribution }}-*.jar /tmp/cve-scan/
+ cp ${{ matrix.scan-path }}/*.jar /tmp/cve-scan/
Review Comment:
so it matches the jars produced by `:iceberg-open-api:stageRESTFixtureRuntime`
##########
docker/iceberg-rest-fixture/Dockerfile:
##########
@@ -30,8 +30,8 @@ RUN set -xeu && \
# Working directory for the application
WORKDIR /usr/lib/iceberg-rest
-# Copy runtime files directly to the target location
-COPY --chown=iceberg:iceberg open-api/build/libs/iceberg-open-api-test-fixtures-runtime-*.jar /usr/lib/iceberg-rest/iceberg-rest-adapter.jar
+# Copy runtime JARs staged by ./gradlew :iceberg-open-api:stageRESTFixtureRuntime
+COPY --chown=iceberg:iceberg open-api/build/rest-fixture-runtime/ /usr/lib/iceberg-rest/libs/
Review Comment:
this is the glue.
moves the jars built by `:iceberg-open-api:stageRESTFixtureRuntime` and saved in `open-api/build/rest-fixture-runtime/ ` to `/usr/lib/iceberg-rest/libs/` which `docker/iceberg-rest-fixture/entrypoint.sh` then reference
##########
build.gradle:
##########
@@ -1219,16 +1216,12 @@ project(':iceberg-open-api') {
}
check.dependsOn('validateRESTCatalogSpec')
- shadowJar {
- archiveBaseName.set("iceberg-open-api-test-fixtures-runtime")
- archiveClassifier.set(null)
- configurations = [project.configurations.testFixturesRuntimeClasspath]
- from sourceSets.testFixtures.output
- zip64 true
-
- manifest {
- attributes 'Main-Class': 'org.apache.iceberg.rest.RESTCatalogServer'
- }
+ tasks.register('stageRESTFixtureRuntime', Sync) {
+ description = 'Stages the REST fixture runtime JARs'
+ group = 'build'
+ into layout.buildDirectory.dir('rest-fixture-runtime')
+ from configurations.testFixturesRuntimeClasspath
+ from tasks.named('testFixturesJar')
Review Comment:
im purposely leaving out pruning deps. will do that in a follow up PR
##########
.github/trivyignore/open-api-test-fixtures-runtime.trivyignore.yaml:
##########
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Trivy ignore file for the OpenAPI test-fixtures runtime.
+#
+vulnerabilities:
Review Comment:
to pass cve scan, same as
https://github.com/apache/iceberg/blob/83bf0ffab77bb45468348fa3f1e9f0e5801e4f62/.github/trivyignore/kafka-connect-runtime.trivyignore.yaml#L20-L25
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stage REST fixture runtime JARs for Docker [iceberg]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#issuecomment-6805296514
Thanks @Fokko, I'll follow up with more PRs to prune what goes into the docker image and then the LICENSE/NOTICE files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stop publishing REST fixture runtime jar [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #16214: Open-API: Stop publishing REST fixture runtime jar
URL: https://github.com/apache/iceberg/pull/16214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stage REST fixture runtime JARs for Docker [iceberg]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu merged PR #16214:
URL: https://github.com/apache/iceberg/pull/16214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Replace shadow JAR with distribution plugin for REST fixture [iceberg]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#issuecomment-4393538356
this is ready for first round of review.
Please take a look at `build.gradle`, `.github/workflows/publish-iceberg-rest-fixture-docker.yml`, `docker/iceberg-rest-fixture/Dockerfile`
and then `open-api/runtime-deps.txt` and `runtime-deps.gradle`
once those are good, please check `open-api/LICENSE` and `open-api/NOTICE`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Replace shadow JAR with distribution plugin for REST fixture [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#discussion_r3269724538
##########
.github/workflows/publish-iceberg-rest-fixture-docker.yml:
##########
@@ -49,8 +49,13 @@ jobs:
distribution: zulu
java-version: 21
- uses: gradle/actions/setup-gradle@0723195856401067f7a2779048b490ace7a47d7c # v5.0.2
- - name: Build Iceberg Open API project
- run: ./gradlew :iceberg-open-api:shadowJar
+ - name: Build REST server and cloud bundle JARs
+ run: |
+ ./gradlew \
+ :iceberg-open-api:installDist \
Review Comment:
Why do we need to build all the shadowJars?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stop publishing REST fixture runtime jar [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#issuecomment-5124830269
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stage REST fixture runtime dependencies for Docker [iceberg]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on code in PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#discussion_r4530486945
##########
.github/workflows/publish-iceberg-rest-fixture-docker.yml:
##########
@@ -49,8 +49,13 @@ jobs:
distribution: zulu
java-version: 21
- uses: gradle/actions/setup-gradle@0723195856401067f7a2779048b490ace7a47d7c # v5.0.2
- - name: Build Iceberg Open API project
- run: ./gradlew :iceberg-open-api:shadowJar
+ - name: Build REST server and cloud bundle JARs
+ run: |
+ ./gradlew \
+ :iceberg-open-api:installDist \
Review Comment:
The workflow invokes only `:iceberg-open-api:stageRESTFixtureRuntime`, not every project’s `shadowJar`. Following the Kafka Connect distribution, the fixture now copies the cloud bundles’ runtime dependency declarations as loose SDK JARs without shipping the shaded cloud bundle JARs. This also fixes the mixed-classpath Azure linkage error found during review. I checked the updated task graph: the only required shadow task is bundled Guava. The OpenAPI shaded runtime task is removed, and staging remains outside the normal OpenAPI build. Backend dependency pruning remains a separate follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stage REST fixture runtime dependencies for Docker [iceberg]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on code in PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#discussion_r4530486945
##########
.github/workflows/publish-iceberg-rest-fixture-docker.yml:
##########
@@ -49,8 +49,13 @@ jobs:
distribution: zulu
java-version: 21
- uses: gradle/actions/setup-gradle@0723195856401067f7a2779048b490ace7a47d7c # v5.0.2
- - name: Build Iceberg Open API project
- run: ./gradlew :iceberg-open-api:shadowJar
+ - name: Build REST server and cloud bundle JARs
+ run: |
+ ./gradlew \
+ :iceberg-open-api:installDist \
Review Comment:
The workflow now invokes only `:iceberg-open-api:stageRESTFixtureRuntime`, so it no longer requests every project's `shadowJar`. I checked the rebased task graph: the remaining shadow tasks are bundled Guava and the AWS, Azure, and GCP bundles, which are dependencies of the existing fixture runtime classpath. The OpenAPI shaded runtime task is removed, and staging stays out of the normal OpenAPI build. Pruning the fixture's dependencies remains a separate follow-up; this change preserves the current backend support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stage REST fixture runtime dependencies for Docker [iceberg]
Posted by "Fokko (via GitHub) via issues" <is...@iceberg.apache.org>.
Fokko commented on code in PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#discussion_r4535880236
##########
docker/iceberg-rest-fixture/entrypoint.sh:
##########
@@ -20,7 +20,8 @@
set -eu
-CLASSPATH="iceberg-rest-adapter.jar"
+# The staged runtime puts all REST server jars under libs/; Java expands this wildcard.
+CLASSPATH="libs/*"
Review Comment:
Is LOG_LEVEL still picked up? The staged libs contain both slf4j-simple and log4j-slf4j-impl (the latter via hadoop-common), and with two SLF4J providers the one with the higher priority wins, which I believe is log4j. That would make the simpleLogger property a no-op. Is this on purpose? Any way to verify with tests that we don't change behavior?
##########
.github/workflows/cve-scan.yml:
##########
@@ -167,9 +167,10 @@ jobs:
scan-path: flink/v2.3/flink-runtime/build/libs
unpack: false
- distribution: open-api-test-fixtures-runtime
- build-task: :iceberg-open-api:shadowJar
- scan-path: open-api/build/libs
+ build-task: :iceberg-open-api:stageRESTFixtureRuntime
+ scan-path: open-api/build/iceberg-rest-server/libs
unpack: false
+ copy-all: true
Review Comment:
This job is red: Trivy flags CVE-2026-102554 (HIGH) in guava 33.4.8 embedded in hadoop-shaded-guava-1.5.0.jar, which comes in via hadoop-common 3.4.3. 1.5.0 is the latest hadoop-shaded-guava release, so there is nothing to bump. Kafka Connect already ignores this exact CVE for the same reason: would the same trivyignore file work here?
```suggestion
build-task: :iceberg-open-api:stageRESTFixtureRuntime
scan-path: open-api/build/iceberg-rest-server/libs
unpack: false
copy-all: true
trivyignores: .github/trivyignore/open-api-test-fixtures-runtime.trivyignore.yaml
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Replace shadow JAR with distribution plugin for REST fixture [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#issuecomment-4394590324
I'm wondering whether we need all those jars and bundles for docker image and integration tests in iceberg-python and iceberg-rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stage REST fixture runtime dependencies for Docker [iceberg]
Posted by "Fokko (via GitHub) via issues" <is...@iceberg.apache.org>.
Fokko commented on code in PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#discussion_r4535880236
##########
docker/iceberg-rest-fixture/entrypoint.sh:
##########
@@ -20,7 +20,8 @@
set -eu
-CLASSPATH="iceberg-rest-adapter.jar"
+# The staged runtime puts all REST server jars under libs/; Java expands this wildcard.
+CLASSPATH="libs/*"
Review Comment:
Is LOG_LEVEL still picked up? The staged libs contain both slf4j-simple and log4j-slf4j-impl (the latter via hadoop-common), and with two SLF4J providers the one with the higher priority wins, which I believe is log4j. That would make the simpleLogger property a no-op. Is this on purpose? Any way to verify with tests that we don't change behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Open-API: Stage REST fixture runtime dependencies for Docker [iceberg]
Posted by "kevinjqliu (via GitHub) via issues" <is...@iceberg.apache.org>.
kevinjqliu commented on code in PR #16214:
URL: https://github.com/apache/iceberg/pull/16214#discussion_r4530486945
##########
.github/workflows/publish-iceberg-rest-fixture-docker.yml:
##########
@@ -49,8 +49,13 @@ jobs:
distribution: zulu
java-version: 21
- uses: gradle/actions/setup-gradle@0723195856401067f7a2779048b490ace7a47d7c # v5.0.2
- - name: Build Iceberg Open API project
- run: ./gradlew :iceberg-open-api:shadowJar
+ - name: Build REST server and cloud bundle JARs
+ run: |
+ ./gradlew \
+ :iceberg-open-api:installDist \
Review Comment:
The workflow invokes only `:iceberg-open-api:stageRESTFixtureRuntime`, not every project’s `shadowJar`. Following the Kafka Connect distribution, the fixture now copies the cloud bundles’ runtime dependency declarations as loose SDK JARs without shipping the shaded cloud bundle JARs. This also fixes the mixed-classpath Azure linkage error found during review. I checked the updated task graph: the only required shadow task is bundled Guava. The OpenAPI shaded runtime task is removed, and staging remains outside the normal OpenAPI build. Backend dependency pruning remains a separate follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org