You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gravitino.apache.org by "nevzheng (via GitHub)" <gi...@apache.org> on 2026/07/08 00:53:44 UTC
[D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
GitHub user nevzheng created a discussion: [DISCUSS] Iceberg V3 type support in Gravitino
### Current state
While working on #11927 / #11928 (the native API failing to load Iceberg tables with a V3 `variant` column), I found that Gravitino's unified type model has no representation for Iceberg's V3 net-new types. With #11928 merged, all of them load through the native API as `ExternalType`:
| Iceberg V3 type | Maps to today | Native Gravitino type? |
|---|---|---|
| `variant` | `external(variant)` | none |
| `geometry` | `external(GEOMETRY)` | none |
| `geography` | `external(GEOGRAPHY)` | none |
| `timestamp_ns` | `external(TIMESTAMP_NANO)` | looks possible — `TimestampType` takes a precision |
| `timestamptz_ns` | `external(TIMESTAMP_NANO)` | looks possible — `TimestampType` takes a precision |
| `unknown` | `external(UNKNOWN)` | maybe `NullType` — defer to maintainers |
### The question
Is this working-as-intended, or do we want native support for these types — and if so, how?
### Decisions we need
1. Is `ExternalType` the intended long-term mapping for Iceberg V3 types, or is this a gap to close?
2. If we want native support — which types, and how do they enter the unified model (enum + JSON serde + DTOs + clients + per-catalog converters)?
### Options to seed the discussion
- **A — Keep V3 types as `ExternalType`.** Document the limitations, defer native support. This is what `ExternalType` was designed for (#3500). But note the downstream cost: `external(...)` columns aren't always consumable — e.g. `external(serial)` failed when queried through Trino (#10957), and the fix there was to give the type a *native* mapping rather than leave it external.
- **B — Implement native support for V3 types.** Add native types (e.g. `VariantType`, geospatial types) to the unified model. These are independent types (variant, geospatial, nanosecond timestamps), so they could be pursued separately / incrementally rather than as one change. `timestamp_ns` in particular looks close — `TimestampType` already carries precision (#2431).
### References
**How the escape hatches were designed**
- #3500 / #3501 — `ExternalType` added for "unsolved catalog type" (its intended purpose).
- #2117 / #2140 — `UnparsedType` added to handle an unresolvable type from the catalog (the alternative fallback).
**Where `ExternalType` has caused problems**
- #10957 / #10970 — `external(serial)` unqueryable via Trino; resolved by mapping to a native type.
- #11805 — `ExternalType.catalogString()` written verbatim into DDL SQL (write-path hazard).
- #11910 / #11915 — ClickHouse types falling through to `ExternalType`.
**Iceberg V3 surface**
- #11927 / #11928 — the `variant` load bug + fix that surfaced this.
- #10994 — variant on the IRC/JDBC backend; noted as under a dedicated community meeting.
- #11293 / #11219 — server-side Iceberg upgraded to 1.11.0 (brought V3 types into range).
- #10995, #10986 — other V3 gaps (deletion vectors; format-version-3 downgraded to 2).
**Timestamp precision precedent**
- #2431 — time/timestamp precision support (why `timestamp_ns` looks representable natively).
GitHub link: https://github.com/apache/gravitino/discussions/11929
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "nevzheng (via GitHub)" <gi...@apache.org>.
GitHub user nevzheng edited a comment on the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
# Proposal: Native `variant` type support in Gravitino
Status: draft / notes
## Summary
Promote Iceberg V3 `variant` from the `ExternalType("variant")` stopgap (#11928) to a
first-class `VariantType` leaf. It stays opaque and just gains a real identity in the
type model. Small change: at the schema level a variant is a parameterless leaf, the
same shape as the existing `UUIDType`.
## What a variant is
A column type for arbitrary, JSON-like semi-structured data whose shape varies row to
row. At the schema level it is the bare, parameterless word `variant`: all structure
lives in the values, not the type. A metadata catalog sees one opaque leaf.
## Current state
Gravitino only records that a column is a variant; it never reads values. The IRC path
already handled it (#10994); the native converter (#11927) threw `UnsupportedOperationException:
Unsupported type: variant`. After #11928 it maps to `ExternalType.of("variant")`, so the
column loads. That is already a relabeled blob, but as a string, not a type: nothing can
branch on it or map it deliberately per connector.
## The change
Serde here is of the schema, not values: a `Type` object round-trips to JSON to cross the
REST API and the entity store. A variant leaf serializes to the bare token `"variant"`
(like `uuid`), so this is four small edits:
- `Type.Name.VARIANT` - new enum tag.
- `Types.VariantType` - singleton leaf extending `PrimitiveType` (a `UUIDType` clone).
- `JsonUtils` - add `VariantType.get()` to the primitive `TYPES` map (read) and a
`case VARIANT` in `writeDataType` (write).
- `FromIcebergType.variant()` - return `Types.VariantType.get()` instead of `ExternalType`.
Out of scope: introspection (no sub-fields, no `variant_get`-style accessors). Those are
query-engine operations on row data, not catalog metadata.
## Decisions and trade-offs
- **Extend `PrimitiveType`** (not `ComplexType`, not raw `Type`): a parameterless childless
leaf fits the primitive serde path with no bespoke code. Risk: "primitive" reads oddly
for semi-structured data, but the base class classifies by structure and `uuid`/`binary`
set the precedent.
- **Iceberg-first**: other connectors keep mapping variant to `ExternalType`. Adding
`Name.VARIANT` touches every `Type.Name` switch (Hive, Glue, Paimon, Lance, Trino,
Flink, JDBC, Spark), but only Iceberg emits one today. Risk: a new constant can reach a
switch `default` elsewhere; audit each and convert connectors individually later.
- **Why not stay external**: `external(serial)` is unqueryable via Trino (#10957) and
`catalogString()` has been written verbatim into DDL SQL unvalidated (#11805). A modeled
identity avoids passing an opaque string through those paths.
## PoC
https://github.com/apache/gravitino/pull/11932
## References
- #11927 (open, bug) - native metadata API fails to load V3 variant columns.
- #11928 (open, PR) - the `ExternalType` stopgap this builds on.
- #10994 (closed) - V3 variant in the Iceberg REST server JDBC backend (the IRC path).
- #3500 (closed) - `ExternalType` for unsolved catalog type (the escape hatch's origin).
- #10957 (open) - `external(serial)` unqueryable via Trino.
- #11805 (open) - `ExternalType.catalogString()` written verbatim into DDL SQL.
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17567285
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "nevzheng (via GitHub) via dev" <de...@gravitino.apache.org>.
GitHub user nevzheng closed a discussion: [DISCUSS] Iceberg V3 type support in Gravitino
### Current state
While working on #11927 / #11928 (the native API failing to load Iceberg tables with a V3 `variant` column), I found that Gravitino's unified type model has no representation for Iceberg's V3 net-new types. With #11928 merged, all of them load through the native API as `ExternalType`:
| Iceberg V3 type | Maps to today | Native Gravitino type? |
|---|---|---|
| `variant` | `external(variant)` | none |
| `geometry` | `external(GEOMETRY)` | none |
| `geography` | `external(GEOGRAPHY)` | none |
| `timestamp_ns` | `external(TIMESTAMP_NANO)` | looks possible — `TimestampType` takes a precision |
| `timestamptz_ns` | `external(TIMESTAMP_NANO)` | looks possible — `TimestampType` takes a precision |
| `unknown` | `external(UNKNOWN)` | maybe `NullType` — defer to maintainers |
### The question
Is this working-as-intended, or do we want native support for these types — and if so, how?
### Decisions we need
1. Is `ExternalType` the intended long-term mapping for Iceberg V3 types, or is this a gap to close?
2. If we want native support — which types, and how do they enter the unified model (enum + JSON serde + DTOs + clients + per-catalog converters)?
### Options to seed the discussion
- **A — Keep V3 types as `ExternalType`.** Document the limitations, defer native support. This is what `ExternalType` was designed for (#3500). But note the downstream cost: `external(...)` columns aren't always consumable — e.g. `external(serial)` failed when queried through Trino (#10957), and the fix there was to give the type a *native* mapping rather than leave it external.
- **B — Implement native support for V3 types.** Add native types (e.g. `VariantType`, geospatial types) to the unified model. These are independent types (variant, geospatial, nanosecond timestamps), so they could be pursued separately / incrementally rather than as one change. `timestamp_ns` in particular looks close — `TimestampType` already carries precision (#2431).
### References
**How the escape hatches were designed**
- #3500 / #3501 — `ExternalType` added for "unsolved catalog type" (its intended purpose).
- #2117 / #2140 — `UnparsedType` added to handle an unresolvable type from the catalog (the alternative fallback).
**Where `ExternalType` has caused problems**
- #10957 / #10970 — `external(serial)` unqueryable via Trino; resolved by mapping to a native type.
- #11805 — `ExternalType.catalogString()` written verbatim into DDL SQL (write-path hazard).
- #11910 / #11915 — ClickHouse types falling through to `ExternalType`.
**Iceberg V3 surface**
- #11927 / #11928 — the `variant` load bug + fix that surfaced this.
- #10994 — variant on the IRC/JDBC backend; noted as under a dedicated community meeting.
- #11293 / #11219 — server-side Iceberg upgraded to 1.11.0 (brought V3 types into range).
- #10995, #10986 — other V3 gaps (deletion vectors; format-version-3 downgraded to 2).
**Timestamp precision precedent**
- #2431 — time/timestamp precision support (why `timestamp_ns` looks representable natively).
---
cc @roryqi @markhoerth — can you help me find the right people to pull into this thread? Thanks!
GitHub link: https://github.com/apache/gravitino/discussions/11929
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "jerryshao (via GitHub)" <gi...@apache.org>.
GitHub user jerryshao added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
I think we should add a built-in variant type support in Gravitino.
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17566207
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "nevzheng (via GitHub)" <gi...@apache.org>.
GitHub user nevzheng added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
Acknowledged — #11949 is now the canonical issue tracking native Iceberg V3 `variant` support, implemented by #11932. Let's focus further discussion there; this thread stays as background. Thanks all!
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17576406
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "nevzheng (via GitHub)" <gi...@apache.org>.
GitHub user nevzheng added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
# Proposal: Native `variant` type support in Gravitino
Status: draft / notes
## Summary
Promote Iceberg V3 `variant` from the `ExternalType("variant")` stopgap (#11928) to a
first-class `VariantType` leaf. It stays opaque and just gains a real identity in the
type model. Small change: at the schema level a variant is a parameterless leaf, the
same shape as the existing `UUIDType`.
## What a variant is
A column type for arbitrary, JSON-like semi-structured data whose shape varies row to
row. At the schema level it is the bare, parameterless word `variant`: all structure
lives in the values, not the type. A metadata catalog sees one opaque leaf.
## Current state
Gravitino only records that a column is a variant; it never reads values. The IRC path
already handled it (#10994); the native converter (#11927) threw `UnsupportedOperationException:
Unsupported type: variant`. After #11928 it maps to `ExternalType.of("variant")`, so the
column loads. That is already a relabeled blob, but as a string, not a type: nothing can
branch on it or map it deliberately per connector.
## The change
Serde here is of the schema, not values: a `Type` object round-trips to JSON to cross the
REST API and the entity store. A variant leaf serializes to the bare token `"variant"`
(like `uuid`), so this is four small edits:
- `Type.Name.VARIANT` - new enum tag.
- `Types.VariantType` - singleton leaf extending `PrimitiveType` (a `UUIDType` clone).
- `JsonUtils` - add `VariantType.get()` to the primitive `TYPES` map (read) and a
`case VARIANT` in `writeDataType` (write).
- `FromIcebergType.variant()` - return `Types.VariantType.get()` instead of `ExternalType`.
Out of scope: introspection (no sub-fields, no `variant_get`-style accessors). Those are
query-engine operations on row data, not catalog metadata.
## Decisions and trade-offs
- **Extend `PrimitiveType`** (not `ComplexType`, not raw `Type`): a parameterless childless
leaf fits the primitive serde path with no bespoke code. Risk: "primitive" reads oddly
for semi-structured data, but the base class classifies by structure and `uuid`/`binary`
set the precedent.
- **Iceberg-first**: other connectors keep mapping variant to `ExternalType`. Adding
`Name.VARIANT` touches every `Type.Name` switch (Hive, Glue, Paimon, Lance, Trino,
Flink, JDBC, Spark), but only Iceberg emits one today. Risk: a new constant can reach a
switch `default` elsewhere; audit each and convert connectors individually later.
- **Why not stay external**: `external(serial)` is unqueryable via Trino (#10957) and
`catalogString()` has been written verbatim into DDL SQL unvalidated (#11805). A modeled
identity avoids passing an opaque string through those paths.
## PoC
todo(nevzheng@): one PR carrying all four edits plus tests (unit serde round-trip + an
Iceberg IT), not split. A standalone core PR would land a `VariantType` no converter
produces yet. Preference, not a requirement; open to splitting core from the Iceberg
wiring if reviewers prefer smaller diffs.
## References
- #11927 (open, bug) - native metadata API fails to load V3 variant columns.
- #11928 (open, PR) - the `ExternalType` stopgap this builds on.
- #10994 (closed) - V3 variant in the Iceberg REST server JDBC backend (the IRC path).
- #3500 (closed) - `ExternalType` for unsolved catalog type (the escape hatch's origin).
- #10957 (open) - `external(serial)` unqueryable via Trino.
- #11805 (open) - `ExternalType.catalogString()` written verbatim into DDL SQL.
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17567285
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "nevzheng (via GitHub) via dev" <de...@gravitino.apache.org>.
GitHub user nevzheng added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
Stepping away from the project, closing as not planned.
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-19417610
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "roryqi (via GitHub)" <gi...@apache.org>.
GitHub user roryqi added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
Maybe we can fix the error types in branch 1.3 using external type. We can add new built-in type in branch 2.0. Just my cent.
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17566218
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "roryqi (via GitHub)" <gi...@apache.org>.
GitHub user roryqi edited a comment on the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
Maybe we can fix the error types in version 1.3 using external type. We can add new built-in type in version 2.0. Just my cent.
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17566218
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "roryqi (via GitHub)" <gi...@apache.org>.
GitHub user roryqi edited a comment on the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
Thanks your proposal. Maybe you write a design document and raise a pull request. You can see https://github.com/apache/gravitino/pull/11852
You can refer to our design document skills https://github.com/apache/gravitino/tree/main/.claude/skills/gravitino-design-doc .
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17567379
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "mchades (via GitHub)" <gi...@apache.org>.
GitHub user mchades added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
I am reviewing PR #11932 and suggest that you create an issue from this discussion to formally track the matter.
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17568694
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "roryqi (via GitHub)" <gi...@apache.org>.
GitHub user roryqi added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
Thanks your proposal. Maybe you write a design document and raise a pull request. You can see https://github.com/apache/gravitino/pull/11852 (Th)
You can refer to our design document skills https://github.com/apache/gravitino/tree/main/.claude/skills/gravitino-design-doc .
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17567379
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "jerryshao (via GitHub)" <gi...@apache.org>.
GitHub user jerryshao added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
@mchades Can you please also take a look? BTW, this is a small feature, I don't think we need a formal design doc for this.
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17567479
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "nevzheng (via GitHub)" <gi...@apache.org>.
GitHub user nevzheng edited a discussion: [DISCUSS] Iceberg V3 type support in Gravitino
### Current state
While working on #11927 / #11928 (the native API failing to load Iceberg tables with a V3 `variant` column), I found that Gravitino's unified type model has no representation for Iceberg's V3 net-new types. With #11928 merged, all of them load through the native API as `ExternalType`:
| Iceberg V3 type | Maps to today | Native Gravitino type? |
|---|---|---|
| `variant` | `external(variant)` | none |
| `geometry` | `external(GEOMETRY)` | none |
| `geography` | `external(GEOGRAPHY)` | none |
| `timestamp_ns` | `external(TIMESTAMP_NANO)` | looks possible — `TimestampType` takes a precision |
| `timestamptz_ns` | `external(TIMESTAMP_NANO)` | looks possible — `TimestampType` takes a precision |
| `unknown` | `external(UNKNOWN)` | maybe `NullType` — defer to maintainers |
### The question
Is this working-as-intended, or do we want native support for these types — and if so, how?
### Decisions we need
1. Is `ExternalType` the intended long-term mapping for Iceberg V3 types, or is this a gap to close?
2. If we want native support — which types, and how do they enter the unified model (enum + JSON serde + DTOs + clients + per-catalog converters)?
### Options to seed the discussion
- **A — Keep V3 types as `ExternalType`.** Document the limitations, defer native support. This is what `ExternalType` was designed for (#3500). But note the downstream cost: `external(...)` columns aren't always consumable — e.g. `external(serial)` failed when queried through Trino (#10957), and the fix there was to give the type a *native* mapping rather than leave it external.
- **B — Implement native support for V3 types.** Add native types (e.g. `VariantType`, geospatial types) to the unified model. These are independent types (variant, geospatial, nanosecond timestamps), so they could be pursued separately / incrementally rather than as one change. `timestamp_ns` in particular looks close — `TimestampType` already carries precision (#2431).
### References
**How the escape hatches were designed**
- #3500 / #3501 — `ExternalType` added for "unsolved catalog type" (its intended purpose).
- #2117 / #2140 — `UnparsedType` added to handle an unresolvable type from the catalog (the alternative fallback).
**Where `ExternalType` has caused problems**
- #10957 / #10970 — `external(serial)` unqueryable via Trino; resolved by mapping to a native type.
- #11805 — `ExternalType.catalogString()` written verbatim into DDL SQL (write-path hazard).
- #11910 / #11915 — ClickHouse types falling through to `ExternalType`.
**Iceberg V3 surface**
- #11927 / #11928 — the `variant` load bug + fix that surfaced this.
- #10994 — variant on the IRC/JDBC backend; noted as under a dedicated community meeting.
- #11293 / #11219 — server-side Iceberg upgraded to 1.11.0 (brought V3 types into range).
- #10995, #10986 — other V3 gaps (deletion vectors; format-version-3 downgraded to 2).
**Timestamp precision precedent**
- #2431 — time/timestamp precision support (why `timestamp_ns` looks representable natively).
---
cc @roryqi @markhoerth — can you help me find the right people to pull into this thread? Thanks!
GitHub link: https://github.com/apache/gravitino/discussions/11929
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "nevzheng (via GitHub)" <gi...@apache.org>.
GitHub user nevzheng added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
- I got the go ahead to own + drive discussion + implementation of Iceberg V3 support. Starting with `variant`.
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17567017
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org
Re: [D] [DISCUSS] Iceberg V3 type support in Gravitino [gravitino]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
GitHub user xxubai added a comment to the discussion: [DISCUSS] Iceberg V3 type support in Gravitino
I can help with the Iceberg V3 adaptation, such as supporting built-in types like Variant :)
GitHub link: https://github.com/apache/gravitino/discussions/11929#discussioncomment-17566811
----
This is an automatically sent email for dev@gravitino.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gravitino.apache.org