You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "c-thiel (via GitHub)" <gi...@apache.org> on 2026/07/17 12:56:20 UTC
[PR] feat(arrow): recognize arrow.parquet.variant on schema import [iceberg-rust]
c-thiel opened a new pull request, #2840:
URL: https://github.com/apache/iceberg-rust/pull/2840
Teach arrow_schema_to_schema to fold a field tagged with the canonical arrow.parquet.variant extension into an atomic Type::Variant, the inverse of the export path added in #2188. Previously such a schema errored with "Field id not found in metadata" because the variant's metadata/value storage sub-fields carry no field id.
## Which issue does this PR close?
PR 1 of 2 for Variant read support.
## What changes are included in this PR?
Add a defaulted `ArrowSchemaVisitor::variant` dispatched via a new `visit_field` helper at each field-dispatch site. The default re-enters normal traversal, so the reconstructing visitors (Int96Coercion, MetadataStrip) are unchanged; `ArrowSchemaConverter` overrides it to fold the struct without descending, guarding that the extension sits on Struct storage. Mirrors Java's Parquet reader, which recovers a variant from its group-level annotation without visiting the group's children.
Covers top-level, struct-nested, list-element and map-value positions plus auto-assigned ids; regenerates the iceberg public-api baseline.
## Are these changes tested?
unit-tested yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(arrow): recognize arrow.parquet.variant on schema import [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY merged PR #2840:
URL: https://github.com/apache/iceberg-rust/pull/2840
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(arrow): recognize arrow.parquet.variant on schema import [iceberg-rust]
Posted by "c-thiel (via GitHub)" <gi...@apache.org>.
c-thiel commented on code in PR #2840:
URL: https://github.com/apache/iceberg-rust/pull/2840#discussion_r3612606807
##########
crates/iceberg/src/arrow/schema.rs:
##########
@@ -544,6 +569,21 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter {
)),
}
}
+
+ fn variant(&mut self, field: &FieldRef) -> Result<Self::T> {
+ // The extension may only sit on struct storage (mirrors
+ // `VariantExtensionType::supports_data_type`).
+ if !matches!(field.data_type(), DataType::Struct(_)) {
Review Comment:
I thought about this too - my main reasons for not matching the nested types are:
1. I believe when we use shredding the storage has typed value fields, not just `{metadata, value}`
1. It mirrors Java (which isn't necessarily a good reason though): `MessageTypeToType.variant()` returns `VariantType.get()` and `ParquetTypeVisitor` short-circuits the group without validating its children; structural checks are deferred to read time.
1. Recognition's job is "is this a Variant?" — the extension answers that. Validating we can actually read the storage (and rejecting shredded for now) belongs in the read PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat(arrow): recognize arrow.parquet.variant on schema import [iceberg-rust]
Posted by "CTTY (via GitHub)" <gi...@apache.org>.
CTTY commented on code in PR #2840:
URL: https://github.com/apache/iceberg-rust/pull/2840#discussion_r3607053882
##########
crates/iceberg/src/arrow/schema.rs:
##########
@@ -164,6 +164,21 @@ pub trait ArrowSchemaVisitor {
/// Called when see a primitive type.
fn primitive(&mut self, p: &DataType) -> Result<Self::T>;
+
+ /// Called when a field carries the `arrow.parquet.variant` extension type.
+ ///
+ /// The default treats the variant as its underlying Arrow struct storage: it
+ /// re-enters normal traversal, preserving the behavior of visitors that don't
+ /// special-case variants. A visitor that produces Iceberg types (or otherwise
+ /// needs variant identity) should override this to fold the struct into a
+ /// single logical variant.
+ ///
+ /// Takes the `&FieldRef` rather than a `&DataType` because the variant signal
+ /// lives on the field's metadata, not on its data type.
Review Comment:
Makes sense
##########
crates/iceberg/src/arrow/schema.rs:
##########
@@ -544,6 +569,21 @@ impl ArrowSchemaVisitor for ArrowSchemaConverter {
)),
}
}
+
+ fn variant(&mut self, field: &FieldRef) -> Result<Self::T> {
+ // The extension may only sit on struct storage (mirrors
+ // `VariantExtensionType::supports_data_type`).
+ if !matches!(field.data_type(), DataType::Struct(_)) {
Review Comment:
Should we match the nested types here as well (metadata and value)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org