You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@xtable.apache.org by "vinishjail97 (via GitHub)" <gi...@apache.org> on 2026/09/25 21:59:38 UTC
[PR] [944] Bundle Iceberg's GlueCatalog and S3FileIO in xtable-utilities [incubator-xtable]
vinishjail97 opened a new pull request, #952:
URL: https://github.com/apache/incubator-xtable/pull/952
Fixes #944
## What is the purpose of the pull request
In 0.3.0 the `xtable-utilities` bundle could load Iceberg's `org.apache.iceberg.aws.glue.GlueCatalog` through `--icebergCatalogConfig` with no extra jars. #822 made `iceberg-hive-runtime` `provided` in `xtable-hive-metastore`, and that jar was the only source of `org.apache.iceberg.aws.*` and `dev.failsafe` in the bundle. The `iceberg-hive-runtime` shade include in `xtable-utilities` stayed, but it no longer matches anything, so the classes dropped out of the 0.4.0 jar.
## Brief change log
- Add `org.apache.iceberg:iceberg-aws` (at `${iceberg.version}`, 1.9.2) to root `dependencyManagement` and to `xtable-utilities`. It brings in `dev.failsafe:failsafe:3.3.2` as a runtime dependency.
- Add both to the `xtable-utilities` shade includes, and remove the stale `iceberg-hive-runtime` include. Re-adding `iceberg-hive-runtime` itself is not an option, since its 1.7.2 Iceberg classes would clash with `iceberg-core` 1.9.2.
- README: note that `GlueCatalog` and `S3FileIO` are bundled, and that other catalog implementations need `java -cp ... org.apache.xtable.utilities.RunSync` instead of `java -jar`.
The AWS SDK clients that `iceberg-aws` needs (`glue`, `s3`, `sts`) are already in the bundle, so `iceberg-aws-bundle` is not needed. The bundled jar grows by about 15 MB (423 MB to 438 MB).
## Verify this pull request
- Built the bundled jar and ran a small program against it alone: `GlueCatalog`, `S3FileIO`, `S3InputStream`, `FailsafeException`, `GlueClient` and `S3Client` all load, and `CatalogUtil.loadCatalog("org.apache.iceberg.aws.glue.GlueCatalog", ...)` initializes (the same call `IcebergTableManager` makes).
- With fake credentials, a Glue `listNamespaces` call and an S3 read through `S3InputStream` (the Failsafe path) both reach AWS and fail only with auth errors, not missing classes.
- `./mvnw -pl xtable-utilities test` and `spotless:check` pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [944] Bundle Iceberg's GlueCatalog and S3FileIO in xtable-utilities [incubator-xtable]
Posted by "kroushan-nit (via GitHub)" <gi...@apache.org>.
kroushan-nit commented on code in PR #952:
URL: https://github.com/apache/incubator-xtable/pull/952#discussion_r4183591165
##########
xtable-utilities/pom.xml:
##########
@@ -633,10 +640,10 @@
<include>org.apache.hudi:hudi-java-client</include>
<include>org.apache.hudi:hudi-timeline-service</include>
<include>org.apache.iceberg:iceberg-api</include>
+ <include>org.apache.iceberg:iceberg-aws</include>
Review Comment:
Second time this allowlist has silently shrunk the bundle — #841 was JOL and SLF4J from the same #822 change, this is `iceberg-aws`.
`release/scripts/validate_bundled_license_texts.py` already reads the bundled jar, the shade `<includes>` and the dependency tree, but it only walks tree → includes (it iterates `tree_coords(tree)` and skips anything not in `includes`). An `<include>` whose artifact has left the resolved graph is invisible to it, which is exactly how `iceberg-hive-runtime` sat here doing nothing.
Could it also assert the reverse — every `<include>` matches something in the tree? That catches the whole class rather than this instance. It would need the License Check job to cover `xtable-utilities`, which it currently skips.
##########
README.md:
##########
@@ -116,6 +116,7 @@ catalogOptions: # all other options are passed through in a map
key1: value1
key2: value2
```
+The bundled jar includes Iceberg's `org.apache.iceberg.aws.glue.GlueCatalog` and `org.apache.iceberg.aws.s3.S3FileIO`. Other catalog implementations must be added to the classpath, which means running with `java -cp xtable-utilities_2.12-0.5.0-SNAPSHOT-bundled.jar:my-catalog.jar org.apache.xtable.utilities.RunSync ...` instead of `java -jar`.
Review Comment:
This reads as part of step 4 (catalog config), but it's about how to run, and the `java -cp` alternative only makes sense against the `java -jar` line right below it. Would it fit better after step 5, next to the existing "The bundled jar includes hadoop dependencies…" sentence?
##########
xtable-utilities/pom.xml:
##########
@@ -200,6 +200,12 @@
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-parquet</artifactId>
</dependency>
+ <!-- GlueCatalog and S3FileIO for icebergCatalogConfig. The AWS SDK clients it needs are
Review Comment:
"already bundled below" holds for `s3` and `sts`, but `glue` isn't declared in this pom — it arrives transitively from `xtable-aws`. Worth saying, since if `xtable-aws` ever narrows that dependency the `software.amazon.awssdk:glue` include goes stale exactly the way `iceberg-hive-runtime` just did, and nothing here would fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [944] Bundle Iceberg's GlueCatalog and S3FileIO in xtable-utilities [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 commented on code in PR #952:
URL: https://github.com/apache/incubator-xtable/pull/952#discussion_r4202374485
##########
xtable-utilities/pom.xml:
##########
@@ -633,10 +640,10 @@
<include>org.apache.hudi:hudi-java-client</include>
<include>org.apache.hudi:hudi-timeline-service</include>
<include>org.apache.iceberg:iceberg-api</include>
+ <include>org.apache.iceberg:iceberg-aws</include>
Review Comment:
Agreed, checking includes → tree as well would catch this whole class rather than one instance. It touches release tooling and the License Check job scope, so it would be good to track it separately. Filed #963.
##########
xtable-utilities/pom.xml:
##########
@@ -200,6 +200,12 @@
<groupId>org.apache.iceberg</groupId>
<artifactId>iceberg-parquet</artifactId>
</dependency>
+ <!-- GlueCatalog and S3FileIO for icebergCatalogConfig. The AWS SDK clients it needs are
Review Comment:
Fixed in 951c05e. The comment now says that `glue` comes transitively from `xtable-aws`, and that its include goes stale if that dependency narrows.
##########
README.md:
##########
@@ -116,6 +116,7 @@ catalogOptions: # all other options are passed through in a map
key1: value1
key2: value2
```
+The bundled jar includes Iceberg's `org.apache.iceberg.aws.glue.GlueCatalog` and `org.apache.iceberg.aws.s3.S3FileIO`. Other catalog implementations must be added to the classpath, which means running with `java -cp xtable-utilities_2.12-0.5.0-SNAPSHOT-bundled.jar:my-catalog.jar org.apache.xtable.utilities.RunSync ...` instead of `java -jar`.
Review Comment:
Fixed in 951c05e. Moved the sentence after step 5, next to the bundled hadoop dependencies note.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [944] Bundle Iceberg's GlueCatalog and S3FileIO in xtable-utilities [incubator-xtable]
Posted by "vinishjail97 (via GitHub)" <gi...@apache.org>.
vinishjail97 merged PR #952:
URL: https://github.com/apache/incubator-xtable/pull/952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@xtable.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org