You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "nssalian (via GitHub)" <gi...@apache.org> on 2026/08/03 19:53:30 UTC
[PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
nssalian opened a new pull request, #17500:
URL: https://github.com/apache/iceberg/pull/17500
## Rationale
[#17256](https://github.com/apache/iceberg/pull/17256) added the `variant` type to the REST catalog spec. This follows up that change with an RCK test.
## Changes
- `CatalogTests`: adds a `supportsVariant()` flag (default `false`); `testCreateTableWithVariantColumn` (variant at top level, list element, and map value; asserts round-trip + format version 3); `testCreateV2TableWithVariantColumnFails` (variant is v3-only).
- `TestRESTCatalog`: enables `supportsVariant()` so the tests run against the REST reference implementation.
- RCK (`RESTCompatibilityKitCatalogTests` / `RESTCompatibilityKitSuite`): adds opt-in `rck.supports-variant` property (default `false`);
- Documentation in `open-api/README.md`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar merged PR #17500:
URL: https://github.com/apache/iceberg/pull/17500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3738515360
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
+ assertThat(TableUtil.formatVersion(loaded))
+ .as("Table with a variant column must be format version 3")
+ .isEqualTo(3);
+ }
+
+ @Test
+ public void testCreateV2TableWithVariantColumnFails() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()), optional(2, "data", Types.VariantType.get()));
+
+ assertThatThrownBy(
+ () ->
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "2")
+ .create())
+ .hasMessageContaining("is not supported until v3");
Review Comment:
To clean up I made it .hasMessageContaining("variant is not supported until v3") and added the tableExists check so it stays portable across catalogs. Client-side pre-rejection feels like a separate RESTSessionCatalog change, happy to file a follow-up if we want it or fold it into this?
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
Review Comment:
Done. The request schema now goes in with ids 10-18 and I assert an explicit expected schema with the reassigned ids 1-9.
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
Review Comment:
Added the struct_data{v: variant} position. good call.
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
Review Comment:
I reframed all three variant guards to .as("Only valid when the catalog supports the variant type").
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3711281303
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -203,6 +204,10 @@ protected boolean supportsEmptyNamespace() {
return false;
}
+ protected boolean supportsVariant() {
+ return false;
Review Comment:
Thanks for adding the feature flag!
As one of those who develop Iceberg REST API, I'd like this to be turned on by default. I expect RCK not to be backward-compatible because it is a test suite. If this is false by default, when another Hive committer updates the Iceberg library, they are unlikely to enable this flag. If this is true by default, the feature is surely tested, or they will explicitly turn off this flag. Either result would help downstream maintainers.
It is not a strong opinion, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "laskoviymishka (via GitHub)" <gi...@apache.org>.
laskoviymishka commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3719090066
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
+ assertThat(TableUtil.formatVersion(loaded))
+ .as("Table with a variant column must be format version 3")
+ .isEqualTo(3);
+ }
+
+ @Test
+ public void testCreateV2TableWithVariantColumnFails() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()), optional(2, "data", Types.VariantType.get()));
+
+ assertThatThrownBy(
+ () ->
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "2")
+ .create())
+ .hasMessageContaining("is not supported until v3");
Review Comment:
This passes for any exception that happens to carry that substring, including an NPE from a path we never meant to reach. Every other `assertThatThrownBy` in this file pins the type first, e.g. `.isInstanceOf(AlreadyExistsException.class).hasMessageContaining(...)`.
Worth being deliberate about which type, though: on the direct path this is `IllegalStateException` from `Schema.checkCompatibility`, but through `RESTCatalogAdapter` that class isn't in `EXCEPTION_ERROR_CODES`, so it comes back as a 500 → `ServiceFailureException` with our original message embedded. That embedding is the only reason the substring matches today.
And `is not supported until v3` is our Java wording — an external RCK server that correctly rejects this with a 400 and its own message would fail here. `tableExists(TBL)` being `false` on the next line is the real conformance invariant, so I'd pin the type and either drop the message match or keep it scoped to the reference run.
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
Review Comment:
Tiny thing, but this description reads as a claim rather than a reason for skipping, which is what shows up in the report when the assumption fails.
The sibling guards phrase it as the condition — `.as("Only valid when the catalog supports nested namespaces")`. I'd match that here and in `testCreateV2TableWithVariantColumnFails`.
##########
open-api/README.md:
##########
@@ -86,6 +86,7 @@ are strictly defined by the REST Specification. The following are currently con
|-------------------------------|---------|
| rck.requires-namespace-create | true |
| rck.supports-serverside-retry | true |
+| rck.supports-variant | false |
Review Comment:
Drive-by while this file is open: the table is also missing `rck.overrides-requested-location` and `rck.supports-names-with-dot`, both `false`. Pre-existing gap and not yours — but it's two rows, and this table is the only place external implementors go looking for these.
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
Review Comment:
This asserts against the same `Schema` object that went into `buildTable`, and the ids here (1–7) are exactly what `assignFreshIds` produces for this shape — so it's passing on an ID coincidence rather than on a stated expectation.
`newTableMetadata` unconditionally reassigns through `TypeUtil.assignFreshIds`, and this file already separates the two cases: `SCHEMA` goes in with ids 3,4 and `TABLE_SCHEMA` with ids 1,2 is what gets asserted, carrying the comment "This is the actual schema for the table, with column IDs reassigned".
I'd mirror that split — pass an input schema with different ids, assert against an explicit expected one. Makes the reassignment part of the contract instead of an accident, and keeps a catalog that assigns ids from a different base from false-failing the kit.
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
Review Comment:
Top-level, list element, and map value cover the interesting positions except variant as a struct field — something like `optional(8, "struct_data", Types.StructType.of(optional(9, "v", Types.VariantType.get())))`.
`checkCompatibility` already walks nested struct fields via `IndexById`, so this isn't a hole in the Java validation. It's a hole in what the RCK tells an external implementor about serializing `struct<v: variant>` through the schema JSON. Cheap to add while we're here — wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3738911005
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
+ assertThat(TableUtil.formatVersion(loaded))
+ .as("Table with a variant column must be format version 3")
+ .isEqualTo(3);
+ }
+
+ @Test
+ public void testCreateV2TableWithVariantColumnFails() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()), optional(2, "data", Types.VariantType.get()));
+
+ assertThatThrownBy(
+ () ->
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "2")
+ .create())
+ .hasMessageContaining("is not supported until v3");
Review Comment:
I think it's fine either way but I do think when we add client-side check, we probably should remove this test. I'm not sure if it's adding too much value for the RCK tests for a given implementation to test this particular not supported case when the client already would prevent the request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3738922085
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
+ assertThat(TableUtil.formatVersion(loaded))
+ .as("Table with a variant column must be format version 3")
+ .isEqualTo(3);
+ }
+
+ @Test
+ public void testCreateV2TableWithVariantColumnFails() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()), optional(2, "data", Types.VariantType.get()));
+
+ assertThatThrownBy(
+ () ->
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "2")
+ .create())
+ .hasMessageContaining("is not supported until v3");
Review Comment:
Ok. Let me remove it to simplify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3724681240
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
Review Comment:
What about a schema evolution case? V3 table without variant and then later adding a variant column?
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
+ assertThat(TableUtil.formatVersion(loaded))
+ .as("Table with a variant column must be format version 3")
+ .isEqualTo(3);
+ }
+
+ @Test
+ public void testCreateV2TableWithVariantColumnFails() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()), optional(2, "data", Types.VariantType.get()));
+
+ assertThatThrownBy(
+ () ->
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "2")
+ .create())
+ .hasMessageContaining("is not supported until v3");
Review Comment:
I'm not sure if this is what @laskoviymishka is saying but fundamentally this test feels a bit weird to me. Is it even hitting the catalog? the validation that's being triggered here is a purely client side validation and so I'm not sure what we're really testing from a compatibility angle here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3737554063
##########
open-api/README.md:
##########
@@ -86,6 +86,7 @@ are strictly defined by the REST Specification. The following are currently con
|-------------------------------|---------|
| rck.requires-namespace-create | true |
| rck.supports-serverside-retry | true |
+| rck.supports-variant | false |
Review Comment:
Will add. Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3724672046
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
+ assertThat(TableUtil.formatVersion(loaded))
+ .as("Table with a variant column must be format version 3")
+ .isEqualTo(3);
+ }
+
+ @Test
+ public void testCreateV2TableWithVariantColumnFails() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()), optional(2, "data", Types.VariantType.get()));
+
+ assertThatThrownBy(
+ () ->
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "2")
+ .create())
+ .hasMessageContaining("is not supported until v3");
Review Comment:
I'm not sure if this is what @laskoviymishka is saying but fundamentally this test feels a bit weird to me. Is it even failing in the catalog? the validation that's being triggered here should just be a purely client side validation and so I'm not sure what we're really testing from a compatibility angle here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#issuecomment-5222437209
Thanks @amogh-jahagirdar. I'll add a follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#issuecomment-5222258570
I'll go ahead and merge this, thanks @nssalian and @okumin @laskoviymishka for reviewing! I do think it would be good to follow up with client side checks; this helps limit the scope of the RCK tests and avoids the whole issue of "What kind of failure message" Does a server return especially when the particular message isn't something that is defined in the spec and as a result we shouldn't really assert on it in the RCK tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3715112161
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -203,6 +204,10 @@ protected boolean supportsEmptyNamespace() {
return false;
}
+ protected boolean supportsVariant() {
+ return false;
Review Comment:
Thank you for taking a look @okumin . I did the following locally:
Ran the RCK downstream: built Iceberg from this branch, pointed apache/hive's metastore-rest-catalog at it, and ran TestRESTCatalogNoneAuth. The two variant tests skip via AssumptionViolatedException: [Catalog supports the variant type], thus proving consumers that don't opt in are unaffected.
On the default: I'd keep it false. It's consistent with the other optional capability flags (supportsNestedNamespaces, supportsServerSideRetry, supportsEmptyNamespace all default
false; only baseline behaviors default true), and variant is a v3 capability that not every implementation may support. Defaulting to true would turn these into failures rather than skips for any existing RCK consumer until they explicitly opt out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3724672046
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
+ assertThat(TableUtil.formatVersion(loaded))
+ .as("Table with a variant column must be format version 3")
+ .isEqualTo(3);
+ }
+
+ @Test
+ public void testCreateV2TableWithVariantColumnFails() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()), optional(2, "data", Types.VariantType.get()));
+
+ assertThatThrownBy(
+ () ->
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "2")
+ .create())
+ .hasMessageContaining("is not supported until v3");
Review Comment:
I'm not sure if this is what @laskoviymishka is saying but fundamentally this test feels a bit weird to me. Is it even hitting the catalog? the validation that's being triggered here should just be a purely client side validation and so I'm not sure what we're really testing from a compatibility angle here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "amogh-jahagirdar (via GitHub)" <gi...@apache.org>.
amogh-jahagirdar commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3724693119
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ assertThat(catalog.tableExists(TBL)).as("Table should not exist").isFalse();
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()),
+ optional(2, "data", Types.VariantType.get()),
+ optional(3, "list_data", Types.ListType.ofOptional(5, Types.VariantType.get())),
+ optional(
+ 4,
+ "map_data",
+ Types.MapType.ofOptional(6, 7, Types.StringType.get(), Types.VariantType.get())));
+
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "3")
+ .create();
+
+ assertThat(catalog.tableExists(TBL)).as("Table should exist").isTrue();
+
+ Table loaded = catalog.loadTable(TBL);
+ assertThat(loaded.schema().asStruct())
+ .as("Variant columns should round-trip through the catalog")
+ .isEqualTo(variantSchema.asStruct());
+ assertThat(TableUtil.formatVersion(loaded))
+ .as("Table with a variant column must be format version 3")
+ .isEqualTo(3);
+ }
+
+ @Test
+ public void testCreateV2TableWithVariantColumnFails() {
+ assumeThat(supportsVariant()).as("Catalog supports the variant type").isTrue();
+
+ C catalog = catalog();
+
+ if (requiresNamespaceCreate()) {
+ catalog.createNamespace(TBL.namespace());
+ }
+
+ Schema variantSchema =
+ new Schema(
+ required(1, "id", Types.LongType.get()), optional(2, "data", Types.VariantType.get()));
+
+ assertThatThrownBy(
+ () ->
+ catalog
+ .buildTable(TBL, variantSchema)
+ .withLocation(baseTableLocation(TBL))
+ .withProperty(TableProperties.FORMAT_VERSION, "2")
+ .create())
+ .hasMessageContaining("is not supported until v3");
Review Comment:
If it is failing in the catalog side, feels like we should just improve the client to detect this case and just reject it before even attempting the table create.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] Core, OpenAPI: Add RCK coverage for variant columns [iceberg]
Posted by "nssalian (via GitHub)" <gi...@apache.org>.
nssalian commented on code in PR #17500:
URL: https://github.com/apache/iceberg/pull/17500#discussion_r3738412616
##########
core/src/test/java/org/apache/iceberg/catalog/CatalogTests.java:
##########
@@ -966,6 +971,71 @@ public void testLoadTable() {
.containsAll(properties.entrySet());
}
+ @Test
+ public void testCreateTableWithVariantColumn() {
Review Comment:
I'll add one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org