You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@polaris.apache.org by "adutra (via GitHub)" <gi...@apache.org> on 2026/05/19 12:51:28 UTC
[PR] Upgrade Iceberg to 1.11.0 [polaris]
adutra opened a new pull request, #4491:
URL: https://github.com/apache/polaris/pull/4491
**This is the first PR for bringing Iceberg 1.11 features into the `main` branch. Once this PR is merged, other features already present in the `feature/iceberg-1.11` will be cherry-picked as new PRs against `main`.**
This change upgrades the Iceberg dependency and REST spec to 1.11.0.
The newly-added test `CatalogTests.testLoadTableWithMissingMetadataFile` is not compatible with REST catalogs, see https://github.com/apache/iceberg/pull/15192. It has been disabled globally.
The newly-added test `CatalogTests.createTableInUniqueLocation` exercises a feature that is not implemented yet. It has been temporarily disabled.
The newly-added test `CatalogTests.dropAfterRenameDoesntCorruptTable` is not compatible with Polaris: it expects the rename-table operation to also move the table base location, which Polaris does not do. It has been disabled globally.
<!--
๐ Describe what changes you're proposing, especially breaking or user-facing changes.
๐ See https://github.com/apache/polaris/blob/main/CONTRIBUTING.md for more.
-->
## Checklist
- [ ] ๐ก๏ธ Don't disclose security issues! (contact security@apache.org)
- [ ] ๐ Clearly explained why the changes are needed, or linked related issues: Fixes #
- [ ] ๐งช Added/updated tests with good coverage, or manually tested (and explained how)
- [ ] ๐ก Added comments for complex logic
- [ ] ๐งพ Updated `CHANGELOG.md` (if needed)
- [ ] ๐ Updated documentation in `site/content/in-dev/unreleased` (if needed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3269545367
##########
plugins/spark/v3.5/spark/src/test/java/org/apache/polaris/spark/SparkCatalogTest.java:
##########
@@ -158,6 +158,7 @@ public void setup() throws Exception {
.thenReturn("/tmp/test-warehouse");
Mockito.when(mockedSession.sparkContext()).thenReturn(mockedContext);
Mockito.when(mockedContext.applicationId()).thenReturn("appId");
+ Mockito.when(mockedContext.appName()).thenReturn("test-app");
Review Comment:
Confirmed: https://github.com/apache/iceberg/pull/14976 added a new mandatory attribute `app-name` to `EnvironmentContext`.
Without this change, all tests in this class fail with:
```
java.lang.NullPointerException
at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
at java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
at org.apache.iceberg.EnvironmentContext.put(EnvironmentContext.java:53)
at org.apache.iceberg.spark.SparkCatalog.initialize(SparkCatalog.java:800)
at org.apache.polaris.spark.SparkCatalogTest$InMemorySparkCatalog.initialize(SparkCatalogTest.java:104)
at org.apache.polaris.spark.SparkCatalogTest.setup(SparkCatalogTest.java:169)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3269907140
##########
plugins/spark/v3.5/spark/src/test/java/org/apache/polaris/spark/SparkCatalogTest.java:
##########
@@ -158,6 +158,7 @@ public void setup() throws Exception {
.thenReturn("/tmp/test-warehouse");
Mockito.when(mockedSession.sparkContext()).thenReturn(mockedContext);
Mockito.when(mockedContext.applicationId()).thenReturn("appId");
+ Mockito.when(mockedContext.appName()).thenReturn("test-app");
Review Comment:
So much for non-breaking changes in a minor version :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3273574624
##########
runtime/service/src/test/java/org/apache/polaris/service/admin/PolarisS3InteroperabilityTest.java:
##########
@@ -134,6 +135,7 @@ private GetNamespaceResponse createNamespace(
.createNamespace(
catalogName,
createNamespaceRequest,
+ new UUID(0L, 0L) /* TODO UUID v7 */,
Review Comment:
Ah - you mean a hardcoded UUID, sorry. Yes, that's certainly possible. Let's tackle that in a follow-up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3269508919
##########
plugins/spark/v3.5/spark/src/test/java/org/apache/polaris/spark/SparkCatalogTest.java:
##########
@@ -158,6 +158,7 @@ public void setup() throws Exception {
.thenReturn("/tmp/test-warehouse");
Mockito.when(mockedSession.sparkContext()).thenReturn(mockedContext);
Mockito.when(mockedContext.applicationId()).thenReturn("appId");
+ Mockito.when(mockedContext.appName()).thenReturn("test-app");
Review Comment:
Some test was failing without this, but I confess I don't remember which one. Let me recheck this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra merged PR #4491:
URL: https://github.com/apache/polaris/pull/4491
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3269913708
##########
runtime/service/src/test/java/org/apache/polaris/service/admin/PolarisS3InteroperabilityTest.java:
##########
@@ -134,6 +135,7 @@ private GetNamespaceResponse createNamespace(
.createNamespace(
catalogName,
createNamespaceRequest,
+ new UUID(0L, 0L) /* TODO UUID v7 */,
Review Comment:
`019e4242-ba18-7d94-b123-758ed2f0fdbf`? (from https://www.uuidgenerator.net/version7)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3269507036
##########
runtime/service/src/test/java/org/apache/polaris/service/admin/PolarisS3InteroperabilityTest.java:
##########
@@ -134,6 +135,7 @@ private GetNamespaceResponse createNamespace(
.createNamespace(
catalogName,
createNamespaceRequest,
+ new UUID(0L, 0L) /* TODO UUID v7 */,
Review Comment:
There is no way to create a UUID v7 with just the JDK. We need a dedicated library for that. I don't want to go down that rabbit hole for now ๐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3266493886
##########
runtime/service/src/test/java/org/apache/polaris/service/catalog/iceberg/AbstractIcebergCatalogViewTest.java:
##########
@@ -276,4 +277,19 @@ public void testEventsAreEmitted() {
afterRefreshEvent.attributes().getRequired(EventAttributes.VIEW_IDENTIFIER))
.isEqualTo(TestData.TABLE);
}
+
+ @Test
+ @Disabled("Polaris REST catalog does not support view registration")
+ @Override
+ public void registerView() {}
Review Comment:
These tests will be re-enabled when we cherry-pick #3689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "adutra (via GitHub)" <gi...@apache.org>.
adutra commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3269545367
##########
plugins/spark/v3.5/spark/src/test/java/org/apache/polaris/spark/SparkCatalogTest.java:
##########
@@ -158,6 +158,7 @@ public void setup() throws Exception {
.thenReturn("/tmp/test-warehouse");
Mockito.when(mockedSession.sparkContext()).thenReturn(mockedContext);
Mockito.when(mockedContext.applicationId()).thenReturn("appId");
+ Mockito.when(mockedContext.appName()).thenReturn("test-app");
Review Comment:
Confirmed: https://github.com/apache/iceberg/pull/14976 added a new mandatory attribute `app-name` to `EnvironmentContext`.
Without this changes all tests in this class fail with:
```
java.lang.NullPointerException
at java.base/java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1011)
at java.base/java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
at org.apache.iceberg.EnvironmentContext.put(EnvironmentContext.java:53)
at org.apache.iceberg.spark.SparkCatalog.initialize(SparkCatalog.java:800)
at org.apache.polaris.spark.SparkCatalogTest$InMemorySparkCatalog.initialize(SparkCatalogTest.java:104)
at org.apache.polaris.spark.SparkCatalogTest.setup(SparkCatalogTest.java:169)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3269925300
##########
runtime/service/src/test/java/org/apache/polaris/service/admin/PolarisS3InteroperabilityTest.java:
##########
@@ -134,6 +135,7 @@ private GetNamespaceResponse createNamespace(
.createNamespace(
catalogName,
createNamespaceRequest,
+ new UUID(0L, 0L) /* TODO UUID v7 */,
Review Comment:
At least it fits the API spec, even though actual idempotency behaviours are not enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3269907140
##########
plugins/spark/v3.5/spark/src/test/java/org/apache/polaris/spark/SparkCatalogTest.java:
##########
@@ -158,6 +158,7 @@ public void setup() throws Exception {
.thenReturn("/tmp/test-warehouse");
Mockito.when(mockedSession.sparkContext()).thenReturn(mockedContext);
Mockito.when(mockedContext.applicationId()).thenReturn("appId");
+ Mockito.when(mockedContext.appName()).thenReturn("test-app");
Review Comment:
So much for non-breaking changes :sweat_smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3269902775
##########
runtime/service/src/test/java/org/apache/polaris/service/admin/PolarisS3InteroperabilityTest.java:
##########
@@ -134,6 +135,7 @@ private GetNamespaceResponse createNamespace(
.createNamespace(
catalogName,
createNamespaceRequest,
+ new UUID(0L, 0L) /* TODO UUID v7 */,
Review Comment:
Well, we could create it externally and have a constant from two longs, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "dimas-b (via GitHub)" <gi...@apache.org>.
dimas-b commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3268922834
##########
runtime/service/src/test/java/org/apache/polaris/service/admin/PolarisS3InteroperabilityTest.java:
##########
@@ -134,6 +135,7 @@ private GetNamespaceResponse createNamespace(
.createNamespace(
catalogName,
createNamespaceRequest,
+ new UUID(0L, 0L) /* TODO UUID v7 */,
Review Comment:
Why not use a fixed UUID v7 now?
##########
plugins/spark/v3.5/spark/src/test/java/org/apache/polaris/spark/SparkCatalogTest.java:
##########
@@ -158,6 +158,7 @@ public void setup() throws Exception {
.thenReturn("/tmp/test-warehouse");
Mockito.when(mockedSession.sparkContext()).thenReturn(mockedContext);
Mockito.when(mockedContext.applicationId()).thenReturn("appId");
+ Mockito.when(mockedContext.appName()).thenReturn("test-app");
Review Comment:
nit: is it required for Iceberg 1.11? Why? ๐ค
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Upgrade Iceberg to 1.11.0 [polaris]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #4491:
URL: https://github.com/apache/polaris/pull/4491#discussion_r3273917786
##########
plugins/spark/v3.5/spark/src/test/java/org/apache/polaris/spark/SparkCatalogTest.java:
##########
@@ -158,6 +158,7 @@ public void setup() throws Exception {
.thenReturn("/tmp/test-warehouse");
Mockito.when(mockedSession.sparkContext()).thenReturn(mockedContext);
Mockito.when(mockedContext.applicationId()).thenReturn("appId");
+ Mockito.when(mockedContext.appName()).thenReturn("test-app");
Review Comment:
This is actually a required key of all spark conf since spark 1.x, we just didn't use it before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@polaris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org