You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by al...@apache.org on 2025/09/08 17:15:51 UTC
(arrow-rs) branch main updated: [Variant] Support Shredded Objects in variant_get (take 2) (#8280)
This is an automated email from the ASF dual-hosted git repository.
alamb pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new fb7d02ed09 [Variant] Support Shredded Objects in variant_get (take 2) (#8280)
fb7d02ed09 is described below
commit fb7d02ed0901bb7fff5fd8e27e52ccc4a28b962c
Author: Ryan Johnson <sc...@users.noreply.github.com>
AuthorDate: Mon Sep 8 10:15:45 2025 -0700
[Variant] Support Shredded Objects in variant_get (take 2) (#8280)
# Which issue does this PR close?
- Closes https://github.com/apache/arrow-rs/issues/8150
- closes https://github.com/apache/arrow-rs/pull/8166
# Rationale for this change
Add support for extracting fields from both shredded and non-shredded
variant arrays at any depth (like "x", "a.x", "a.b.x") and casting them
to Int32 with proper NULL handling for type mismatches.
NOTE: This is a second attempt at
* https://github.com/apache/arrow-rs/pull/8166
which suffered strong logical conflicts with
* https://github.com/apache/arrow-rs/pull/8179
and which itself grew out of
* https://github.com/apache/arrow-rs/pull/8122
See the other two PR for the vast majority of review commentary relating
to this change.
I started from the original PR commits (first three), performed the
merge, and fixed up a bunch of issues.
Manually diffing the before
([76b75eebc..882aa4d69](https://github.com/apache/arrow-rs/compare/76b75eebc..882aa4d69))
and after
([0ba91aed9..f6fd91583](https://github.com/apache/arrow-rs/compare/0ba91aed9..f6fd91583))
diffs gives the following non-trivial differences vs. the original PR
* Ran `cargo fmt`
* `typed_value_to_variant` now supports all primitive numeric types
(previously only int16)
* cast options plumbed through and respected
* Fix a null buffer bug in `shredded_get_path` -- the original code was
wrongly unioning in the null buffer from `typed_value` column:
```patch
// Path exhausted! Create a new `VariantArray` for the location we
landed on.
- // Also union nulls from the final typed_value field we landed on
- if let Some(typed_value) = shredding_state.typed_value_field() {
- accumulated_nulls = arrow::buffer::NullBuffer::union(
- accumulated_nulls.as_ref(),
- typed_value.nulls(),
- );
- }
let target = make_target_variant(
shredding_state.value_field().cloned(),
shredding_state.typed_value_field().cloned(),
accumulated_nulls,
);
```
* Remove the `get_variant_perfectly_shredded_int32_as_variant` test
case, because https://github.com/apache/arrow-rs/pull/8179 introduced a
battery of unit tests that cover the same functionality.
* Remove now-unnecessary `.unwrap()` calls from object builder `finish`
calls in unit tests
* Fixed broken test code in `create_depth_1_shredded_test_data_working`,
which captured the return value of a nested builder's `finish` (`()`)
instead of the return value of the top-level builder. I'm not quite sure
what this code was trying to do, but I changed it to just create a
nested builder instead of a second top-level builder:
```patch
fn create_depth_1_shredded_test_data_working() -> ArrayRef {
// Create metadata following the working pattern from
shredded_object_with_x_field_variant_array
let (metadata, _) = {
- let a_variant = {
- let mut a_builder = parquet_variant::VariantBuilder::new();
- let mut a_obj = a_builder.new_object();
- a_obj.insert("x", Variant::Int32(55)); // "a.x" field (shredded when
possible)
- a_obj.finish().unwrap()
- };
let mut builder = parquet_variant::VariantBuilder::new();
let mut obj = builder.new_object();
- obj.insert("a", a_variant);
+
+ // Create the nested "a" object
+ let mut a_obj = obj.new_object("a");
+ a_obj.insert("x", Variant::Int32(55));
+ a_obj.finish();
+
obj.finish().unwrap();
builder.finish()
};
```
* Similar fix (twice, `a_variant` and `b_variant`) for
`create_depth_2_shredded_test_data_working`
* `make_shredding_row_builder` now supports signed int and float types
(unsigned int not supported yet)
* A new `get_type_name` helper in row_builder.rs that gives
human-readable data type names. I'm not convinced it's necessary (and
the code is in the wrong spot, jammed in the middle of
`VariantAsPrimitive` code.
* `impl VariantAsPrimitive` for all signed int and float types
* `PrimitiveVariantShreddingRowBuilder` now has a lifetime param because
it takes a reference to cast options (it now respects unsafe vs. safe
casting)
# What changes are included in this PR?
Everything in the original PR, plus merge in the main branch, fix
logical conflicts and fix various broken tests.
# Are these changes tested?
All unit tests now pass.
# Are there any user-facing changes?
No (variant is not public yet)
---------
Co-authored-by: carpecodeum <ad...@gmail.com>
Co-authored-by: Andrew Lamb <an...@nerdnetworks.org>
---
parquet-variant-compute/src/variant_array.rs | 341 +++++-
parquet-variant-compute/src/variant_get/mod.rs | 1281 +++++++++++++++++++-
.../src/variant_get/output/mod.rs | 83 +-
.../src/variant_get/output/row_builder.rs | 342 ++++++
parquet-variant/src/path.rs | 10 +-
5 files changed, 1911 insertions(+), 146 deletions(-)
diff --git a/parquet-variant-compute/src/variant_array.rs b/parquet-variant-compute/src/variant_array.rs
index 10fb5f67ee..17b0adbdd0 100644
--- a/parquet-variant-compute/src/variant_array.rs
+++ b/parquet-variant-compute/src/variant_array.rs
@@ -19,8 +19,11 @@
use arrow::array::{Array, ArrayData, ArrayRef, AsArray, BinaryViewArray, StructArray};
use arrow::buffer::NullBuffer;
-use arrow::datatypes::{Int16Type, Int32Type};
-use arrow_schema::{ArrowError, DataType};
+use arrow::datatypes::{
+ Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
+ UInt32Type, UInt64Type, UInt8Type,
+};
+use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
@@ -50,6 +53,9 @@ pub struct VariantArray {
/// Reference to the underlying StructArray
inner: StructArray,
+ /// The metadata column of this variant
+ metadata: BinaryViewArray,
+
/// how is this variant array shredded?
shredding_state: ShreddingState,
}
@@ -104,31 +110,57 @@ impl VariantArray {
)));
};
- // Find the value field, if present
- let value = inner
- .column_by_name("value")
- .map(|v| {
- v.as_binary_view_opt().ok_or_else(|| {
- ArrowError::NotYetImplemented(format!(
- "VariantArray 'value' field must be BinaryView, got {}",
- v.data_type()
- ))
- })
- })
- .transpose()?;
-
- // Find the typed_value field, if present
- let typed_value = inner.column_by_name("typed_value");
+ // Extract value and typed_value fields
+ let value = if let Some(value_col) = inner.column_by_name("value") {
+ if let Some(binary_view) = value_col.as_binary_view_opt() {
+ Some(binary_view.clone())
+ } else {
+ return Err(ArrowError::NotYetImplemented(format!(
+ "VariantArray 'value' field must be BinaryView, got {}",
+ value_col.data_type()
+ )));
+ }
+ } else {
+ None
+ };
+ let typed_value = inner.column_by_name("typed_value").cloned();
// Note these clones are cheap, they just bump the ref count
- let inner = inner.clone();
- let shredding_state =
- ShreddingState::try_new(metadata.clone(), value.cloned(), typed_value.cloned())?;
-
Ok(Self {
+ inner: inner.clone(),
+ metadata: metadata.clone(),
+ shredding_state: ShreddingState::try_new(metadata.clone(), value, typed_value)?,
+ })
+ }
+
+ pub(crate) fn from_parts(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ nulls: Option<NullBuffer>,
+ ) -> Self {
+ let mut builder =
+ StructArrayBuilder::new().with_field("metadata", Arc::new(metadata.clone()));
+ if let Some(value) = value.clone() {
+ builder = builder.with_field("value", Arc::new(value));
+ }
+ if let Some(typed_value) = typed_value.clone() {
+ builder = builder.with_field("typed_value", typed_value);
+ }
+ if let Some(nulls) = nulls {
+ builder = builder.with_nulls(nulls);
+ }
+
+ // This would be a lot simpler if ShreddingState were just a pair of Option... we already
+ // have everything we need.
+ let inner = builder.build();
+ let shredding_state =
+ ShreddingState::try_new(metadata.clone(), value, typed_value).unwrap(); // valid by construction
+ Self {
inner,
+ metadata,
shredding_state,
- })
+ }
}
/// Returns a reference to the underlying [`StructArray`].
@@ -168,10 +200,12 @@ impl VariantArray {
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant<'_, '_> {
match &self.shredding_state {
- ShreddingState::Unshredded { metadata, value } => {
- Variant::new(metadata.value(index), value.value(index))
+ ShreddingState::Unshredded { value, .. } => {
+ // Unshredded case
+ Variant::new(self.metadata.value(index), value.value(index))
}
ShreddingState::Typed { typed_value, .. } => {
+ // Typed case (formerly PerfectlyShredded)
if typed_value.is_null(index) {
Variant::Null
} else {
@@ -179,17 +213,17 @@ impl VariantArray {
}
}
ShreddingState::PartiallyShredded {
- metadata,
- value,
- typed_value,
+ value, typed_value, ..
} => {
+ // PartiallyShredded case (formerly ImperfectlyShredded)
if typed_value.is_null(index) {
- Variant::new(metadata.value(index), value.value(index))
+ Variant::new(self.metadata.value(index), value.value(index))
} else {
typed_value_to_variant(typed_value, index)
}
}
ShreddingState::AllNull { .. } => {
+ // AllNull case: neither value nor typed_value fields exist
// NOTE: This handles the case where neither value nor typed_value fields exist.
// For top-level variants, this returns Variant::Null (JSON null).
// For shredded object fields, this technically should indicate SQL NULL,
@@ -201,7 +235,7 @@ impl VariantArray {
/// Return a reference to the metadata field of the [`StructArray`]
pub fn metadata_field(&self) -> &BinaryViewArray {
- self.shredding_state.metadata_field()
+ &self.metadata
}
/// Return a reference to the value field of the `StructArray`
@@ -215,6 +249,168 @@ impl VariantArray {
}
}
+/// One shredded field of a partially or prefectly shredded variant. For example, suppose the
+/// shredding schema for variant `v` treats it as an object with a single field `a`, where `a` is
+/// itself a struct with the single field `b` of type INT. Then the physical layout of the column
+/// is:
+///
+/// ```text
+/// v: VARIANT {
+/// metadata: BINARY,
+/// value: BINARY,
+/// typed_value: STRUCT {
+/// a: SHREDDED_VARIANT_FIELD {
+/// value: BINARY,
+/// typed_value: STRUCT {
+/// a: SHREDDED_VARIANT_FIELD {
+/// value: BINARY,
+/// typed_value: INT,
+/// },
+/// },
+/// },
+/// },
+/// }
+/// ```
+///
+/// In the above, each row of `v.value` is either a variant value (shredding failed, `v` was not an
+/// object at all) or a variant object (partial shredding, `v` was an object but included unexpected
+/// fields other than `a`), or is NULL (perfect shredding, `v` was an object containing only the
+/// single expected field `a`).
+///
+/// A similar story unfolds for each `v.typed_value.a.value` -- a variant value if shredding failed
+/// (`v:a` was not an object at all), or a variant object (`v:a` was an object with unexpected
+/// additional fields), or NULL (`v:a` was an object containing only the single expected field `b`).
+///
+/// Finally, `v.typed_value.a.typed_value.b.value` is either NULL (`v:a.b` was an integer) or else a
+/// variant value (which could be `Variant::Null`).
+#[derive(Debug)]
+pub struct ShreddedVariantFieldArray {
+ /// Reference to the underlying StructArray
+ inner: StructArray,
+ shredding_state: ShreddingState,
+}
+
+#[allow(unused)]
+impl ShreddedVariantFieldArray {
+ /// Creates a new `ShreddedVariantFieldArray` from a [`StructArray`].
+ ///
+ /// # Arguments
+ /// - `inner` - The underlying [`StructArray`] that contains the variant data.
+ ///
+ /// # Returns
+ /// - A new instance of `ShreddedVariantFieldArray`.
+ ///
+ /// # Errors:
+ /// - If the `StructArray` does not contain the required fields
+ ///
+ /// # Requirements of the `StructArray`
+ ///
+ /// 1. An optional field named `value` that is binary, large_binary, or
+ /// binary_view
+ ///
+ /// 2. An optional field named `typed_value` which can be any primitive type
+ /// or be a list, large_list, list_view or struct
+ ///
+ /// Currently, only `value` columns of type [`BinaryViewArray`] are supported.
+ pub fn try_new(inner: ArrayRef) -> Result<Self, ArrowError> {
+ let Some(inner_struct) = inner.as_struct_opt() else {
+ return Err(ArrowError::InvalidArgumentError(
+ "Invalid ShreddedVariantFieldArray: requires StructArray as input".to_string(),
+ ));
+ };
+
+ // Extract value and typed_value fields (metadata is not expected in ShreddedVariantFieldArray)
+ let value = inner_struct
+ .column_by_name("value")
+ .and_then(|col| col.as_binary_view_opt().cloned());
+ let typed_value = inner_struct.column_by_name("typed_value").cloned();
+
+ // Use a dummy metadata for the constructor (ShreddedVariantFieldArray doesn't have metadata)
+ let dummy_metadata = arrow::array::BinaryViewArray::new_null(inner_struct.len());
+
+ // Note this clone is cheap, it just bumps the ref count
+ let inner = inner_struct.clone();
+ Ok(Self {
+ inner: inner.clone(),
+ shredding_state: ShreddingState::try_new(dummy_metadata, value, typed_value)?,
+ })
+ }
+
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
+ }
+
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
+ }
+
+ /// Returns a reference to the underlying [`StructArray`].
+ pub fn inner(&self) -> &StructArray {
+ &self.inner
+ }
+}
+
+impl Array for ShreddedVariantFieldArray {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn to_data(&self) -> ArrayData {
+ self.inner.to_data()
+ }
+
+ fn into_data(self) -> ArrayData {
+ self.inner.into_data()
+ }
+
+ fn data_type(&self) -> &DataType {
+ self.inner.data_type()
+ }
+
+ fn slice(&self, offset: usize, length: usize) -> ArrayRef {
+ let inner = self.inner.slice(offset, length);
+ let shredding_state = self.shredding_state.slice(offset, length);
+ Arc::new(Self {
+ inner,
+ shredding_state,
+ })
+ }
+
+ fn len(&self) -> usize {
+ self.inner.len()
+ }
+
+ fn is_empty(&self) -> bool {
+ self.inner.is_empty()
+ }
+
+ fn offset(&self) -> usize {
+ self.inner.offset()
+ }
+
+ fn nulls(&self) -> Option<&NullBuffer> {
+ // According to the shredding spec, ShreddedVariantFieldArray should be
+ // physically non-nullable - SQL NULL is inferred by both value and
+ // typed_value being physically NULL
+ None
+ }
+
+ fn get_buffer_memory_size(&self) -> usize {
+ self.inner.get_buffer_memory_size()
+ }
+
+ fn get_array_memory_size(&self) -> usize {
+ self.inner.get_array_memory_size()
+ }
+}
+
/// Represents the shredding state of a [`VariantArray`]
///
/// [`VariantArray`]s can be shredded according to the [Parquet Variant
@@ -246,12 +442,16 @@ pub enum ShreddingState {
metadata: BinaryViewArray,
typed_value: ArrayRef,
},
- /// Partially shredded:
- /// * value is an object
- /// * typed_value is a shredded object.
+ /// Imperfectly shredded: Shredded values reside in `typed_value` while those that failed to
+ /// shred reside in `value`. Missing field values are NULL in both columns, while NULL primitive
+ /// values have NULL `typed_value` and `Variant::Null` in `value`.
///
- /// Note the spec says "Writers must not produce data where both value and
- /// typed_value are non-null, unless the Variant value is an object."
+ /// NOTE: A partially shredded struct is a special kind of imperfect shredding, where
+ /// `typed_value` and `value` are both non-NULL. The `typed_value` is a struct containing the
+ /// subset of fields for which shredding was attempted (each field will then have its own value
+ /// and/or typed_value sub-fields that indicate how shredding actually turned out). Meanwhile,
+ /// the `value` is a variant object containing the subset of fields for which shredding was
+ /// not even attempted.
PartiallyShredded {
metadata: BinaryViewArray,
value: BinaryViewArray,
@@ -348,15 +548,81 @@ impl ShreddingState {
}
}
+/// Builds struct arrays from component fields
+///
+/// TODO: move to arrow crate
+#[derive(Debug, Default, Clone)]
+pub(crate) struct StructArrayBuilder {
+ fields: Vec<FieldRef>,
+ arrays: Vec<ArrayRef>,
+ nulls: Option<NullBuffer>,
+}
+
+impl StructArrayBuilder {
+ pub fn new() -> Self {
+ Default::default()
+ }
+
+ /// Add an array to this struct array as a field with the specified name.
+ pub fn with_field(mut self, field_name: &str, array: ArrayRef) -> Self {
+ let field = Field::new(field_name, array.data_type().clone(), true);
+ self.fields.push(Arc::new(field));
+ self.arrays.push(array);
+ self
+ }
+
+ /// Set the null buffer for this struct array.
+ pub fn with_nulls(mut self, nulls: NullBuffer) -> Self {
+ self.nulls = Some(nulls);
+ self
+ }
+
+ pub fn build(self) -> StructArray {
+ let Self {
+ fields,
+ arrays,
+ nulls,
+ } = self;
+ StructArray::new(Fields::from(fields), arrays, nulls)
+ }
+}
+
/// returns the non-null element at index as a Variant
fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant<'_, '_> {
match typed_value.data_type() {
- DataType::Int32 => {
- primitive_conversion_single_value!(Int32Type, typed_value, index)
+ DataType::Int8 => {
+ primitive_conversion_single_value!(Int8Type, typed_value, index)
}
DataType::Int16 => {
primitive_conversion_single_value!(Int16Type, typed_value, index)
}
+ DataType::Int32 => {
+ primitive_conversion_single_value!(Int32Type, typed_value, index)
+ }
+ DataType::Int64 => {
+ primitive_conversion_single_value!(Int64Type, typed_value, index)
+ }
+ DataType::UInt8 => {
+ primitive_conversion_single_value!(UInt8Type, typed_value, index)
+ }
+ DataType::UInt16 => {
+ primitive_conversion_single_value!(UInt16Type, typed_value, index)
+ }
+ DataType::UInt32 => {
+ primitive_conversion_single_value!(UInt32Type, typed_value, index)
+ }
+ DataType::UInt64 => {
+ primitive_conversion_single_value!(UInt64Type, typed_value, index)
+ }
+ DataType::Float16 => {
+ primitive_conversion_single_value!(Float16Type, typed_value, index)
+ }
+ DataType::Float32 => {
+ primitive_conversion_single_value!(Float32Type, typed_value, index)
+ }
+ DataType::Float64 => {
+ primitive_conversion_single_value!(Float64Type, typed_value, index)
+ }
// todo other types here (note this is very similar to cast_to_variant.rs)
// so it would be great to figure out how to share this code
_ => {
@@ -392,9 +658,11 @@ impl Array for VariantArray {
fn slice(&self, offset: usize, length: usize) -> ArrayRef {
let inner = self.inner.slice(offset, length);
+ let metadata = self.metadata.slice(offset, length);
let shredding_state = self.shredding_state.slice(offset, length);
Arc::new(Self {
inner,
+ metadata,
shredding_state,
})
}
@@ -526,6 +794,7 @@ mod test {
let metadata = BinaryViewArray::from(vec![b"test" as &[u8]]);
let shredding_state = ShreddingState::try_new(metadata.clone(), None, None).unwrap();
+ // Verify the shredding state is AllNull
assert!(matches!(shredding_state, ShreddingState::AllNull { .. }));
// Verify metadata is preserved correctly
diff --git a/parquet-variant-compute/src/variant_get/mod.rs b/parquet-variant-compute/src/variant_get/mod.rs
index 585c4462c3..10403b1369 100644
--- a/parquet-variant-compute/src/variant_get/mod.rs
+++ b/parquet-variant-compute/src/variant_get/mod.rs
@@ -15,25 +15,228 @@
// specific language governing permissions and limitations
// under the License.
use arrow::{
- array::{Array, ArrayRef},
+ array::{self, Array, ArrayRef, BinaryViewArray, StructArray},
compute::CastOptions,
+ datatypes::Field,
error::Result,
};
-use arrow_schema::{ArrowError, FieldRef};
-use parquet_variant::VariantPath;
+use arrow_schema::{ArrowError, DataType, FieldRef};
+use parquet_variant::{VariantPath, VariantPathElement};
use crate::variant_array::ShreddingState;
-use crate::variant_get::output::instantiate_output_builder;
-use crate::VariantArray;
+use crate::{variant_array::ShreddedVariantFieldArray, VariantArray};
+
+use std::sync::Arc;
mod output;
+pub(crate) enum ShreddedPathStep<'a> {
+ /// Path step succeeded, return the new shredding state
+ Success(&'a ShreddingState),
+ /// The path element is not present in the `typed_value` column and there is no `value` column,
+ /// so we we know it does not exist. It, and all paths under it, are all-NULL.
+ Missing,
+ /// The path element is not present in the `typed_value` column and must be retrieved from the `value`
+ /// column instead. The caller should be prepared to handle any value, including the requested
+ /// type, an arbitrary "wrong" type, or `Variant::Null`.
+ NotShredded,
+}
+
+/// Given a shredded variant field -- a `(value?, typed_value?)` pair -- try to take one path step
+/// deeper. For a `VariantPathElement::Field`, the step fails if there is no `typed_value` at this
+/// level, or if `typed_value` is not a struct, or if the requested field name does not exist.
+///
+/// TODO: Support `VariantPathElement::Index`? It wouldn't be easy, and maybe not even possible.
+pub(crate) fn follow_shredded_path_element<'a>(
+ shredding_state: &'a ShreddingState,
+ path_element: &VariantPathElement<'_>,
+ cast_options: &CastOptions,
+) -> Result<ShreddedPathStep<'a>> {
+ // If the requested path element is not present in `typed_value`, and `value` is missing, then
+ // we know it does not exist; it, and all paths under it, are all-NULL.
+ let missing_path_step = || {
+ let Some(_value_field) = shredding_state.value_field() else {
+ return ShreddedPathStep::Missing;
+ };
+ ShreddedPathStep::NotShredded
+ };
+
+ let Some(typed_value) = shredding_state.typed_value_field() else {
+ return Ok(missing_path_step());
+ };
+
+ match path_element {
+ VariantPathElement::Field { name } => {
+ // Try to step into the requested field name of a struct.
+ // First, try to downcast to StructArray
+ let Some(struct_array) = typed_value.as_any().downcast_ref::<StructArray>() else {
+ // Downcast failure - if strict cast options are enabled, this should be an error
+ if !cast_options.safe {
+ return Err(ArrowError::CastError(format!(
+ "Cannot access field '{}' on non-struct type: {}",
+ name,
+ typed_value.data_type()
+ )));
+ }
+ // With safe cast options, return NULL (missing_path_step)
+ return Ok(missing_path_step());
+ };
+
+ // Now try to find the column - missing column in a present struct is just missing data
+ let Some(field) = struct_array.column_by_name(name) else {
+ // Missing column in a present struct is just missing, not wrong - return Ok
+ return Ok(missing_path_step());
+ };
+
+ let field = field
+ .as_any()
+ .downcast_ref::<ShreddedVariantFieldArray>()
+ .ok_or_else(|| {
+ // TODO: Should we blow up? Or just end the traversal and let the normal
+ // variant pathing code sort out the mess that it must anyway be
+ // prepared to handle?
+ ArrowError::InvalidArgumentError(format!(
+ "Expected a ShreddedVariantFieldArray, got {:?} instead",
+ field.data_type(),
+ ))
+ })?;
+
+ Ok(ShreddedPathStep::Success(field.shredding_state()))
+ }
+ VariantPathElement::Index { .. } => {
+ // TODO: Support array indexing. Among other things, it will require slicing not
+ // only the array we have here, but also the corresponding metadata and null masks.
+ Err(ArrowError::NotYetImplemented(
+ "Pathing into shredded variant array index".into(),
+ ))
+ }
+ }
+}
+
+/// Follows the given path as far as possible through shredded variant fields. If the path ends on a
+/// shredded field, return it directly. Otherwise, use a row shredder to follow the rest of the path
+/// and extract the requested value on a per-row basis.
+fn shredded_get_path(
+ input: &VariantArray,
+ path: &[VariantPathElement<'_>],
+ as_field: Option<&Field>,
+ cast_options: &CastOptions,
+) -> Result<ArrayRef> {
+ // Helper that creates a new VariantArray from the given nested value and typed_value columns,
+ // properly accounting for accumulated nulls from path traversal
+ let make_target_variant =
+ |value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ accumulated_nulls: Option<arrow::buffer::NullBuffer>| {
+ let metadata = input.metadata_field().clone();
+ VariantArray::from_parts(metadata, value, typed_value, accumulated_nulls)
+ };
+
+ // Helper that shreds a VariantArray to a specific type.
+ let shred_basic_variant =
+ |target: VariantArray, path: VariantPath<'_>, as_field: Option<&Field>| {
+ let as_type = as_field.map(|f| f.data_type());
+ let mut builder =
+ output::row_builder::make_shredding_row_builder(path, as_type, cast_options)?;
+ for i in 0..target.len() {
+ if target.is_null(i) {
+ builder.append_null()?;
+ } else {
+ builder.append_value(&target.value(i))?;
+ }
+ }
+ builder.finish()
+ };
+
+ // Peel away the prefix of path elements that traverses the shredded parts of this variant
+ // column. Shredding will traverse the rest of the path on a per-row basis.
+ let mut shredding_state = input.shredding_state();
+ let mut accumulated_nulls = input.inner().nulls().cloned();
+ let mut path_index = 0;
+ for path_element in path {
+ match follow_shredded_path_element(shredding_state, path_element, cast_options)? {
+ ShreddedPathStep::Success(state) => {
+ // Union nulls from the typed_value we just accessed
+ if let Some(typed_value) = shredding_state.typed_value_field() {
+ accumulated_nulls = arrow::buffer::NullBuffer::union(
+ accumulated_nulls.as_ref(),
+ typed_value.nulls(),
+ );
+ }
+ shredding_state = state;
+ path_index += 1;
+ continue;
+ }
+ ShreddedPathStep::Missing => {
+ let num_rows = input.len();
+ let arr = match as_field.map(|f| f.data_type()) {
+ Some(data_type) => Arc::new(array::new_null_array(data_type, num_rows)) as _,
+ None => Arc::new(array::NullArray::new(num_rows)) as _,
+ };
+ return Ok(arr);
+ }
+ ShreddedPathStep::NotShredded => {
+ let target = make_target_variant(
+ shredding_state.value_field().cloned(),
+ None,
+ accumulated_nulls,
+ );
+ return shred_basic_variant(target, path[path_index..].into(), as_field);
+ }
+ };
+ }
+
+ // Path exhausted! Create a new `VariantArray` for the location we landed on.
+ let target = make_target_variant(
+ shredding_state.value_field().cloned(),
+ shredding_state.typed_value_field().cloned(),
+ accumulated_nulls,
+ );
+
+ // If our caller did not request any specific type, we can just return whatever we landed on.
+ let Some(as_field) = as_field else {
+ return Ok(Arc::new(target));
+ };
+
+ // Structs are special. Recurse into each field separately, hoping to follow the shredding even
+ // further, and build up the final struct from those individually shredded results.
+ if let DataType::Struct(fields) = as_field.data_type() {
+ let children = fields
+ .iter()
+ .map(|field| {
+ shredded_get_path(
+ &target,
+ &[VariantPathElement::from(field.name().as_str())],
+ Some(field),
+ cast_options,
+ )
+ })
+ .collect::<Result<Vec<_>>>()?;
+
+ let struct_nulls = target.nulls().cloned();
+
+ return Ok(Arc::new(StructArray::try_new(
+ fields.clone(),
+ children,
+ struct_nulls,
+ )?));
+ }
+
+ // Not a struct, so directly shred the variant as the requested type
+ shred_basic_variant(target, VariantPath::default(), Some(as_field))
+}
+
/// Returns an array with the specified path extracted from the variant values.
///
/// The return array type depends on the `as_type` field of the options parameter
/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
/// to the specified path.
/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+///
+/// TODO: How would a caller request a struct or list type where the fields/elements can be any
+/// variant? Caller can pass None as the requested type to fetch a specific path, but it would
+/// quickly become annoying (and inefficient) to call `variant_get` for each leaf value in a struct or
+/// list and then try to assemble the results.
pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
ArrowError::InvalidArgumentError(
@@ -41,25 +244,13 @@ pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
)
})?;
- // Create the output writer based on the specified output options
- let output_builder = instantiate_output_builder(options.clone())?;
-
- // Dispatch based on the shredding state of the input variant array
- match variant_array.shredding_state() {
- ShreddingState::PartiallyShredded {
- metadata,
- value,
- typed_value,
- } => output_builder.partially_shredded(variant_array, metadata, value, typed_value),
- ShreddingState::Typed {
- metadata,
- typed_value,
- } => output_builder.typed(variant_array, metadata, typed_value),
- ShreddingState::Unshredded { metadata, value } => {
- output_builder.unshredded(variant_array, metadata, value)
- }
- ShreddingState::AllNull { metadata } => output_builder.all_null(variant_array, metadata),
- }
+ let GetOptions {
+ as_type,
+ path,
+ cast_options,
+ } = options;
+
+ shredded_get_path(variant_array, &path, as_type.as_deref(), &cast_options)
}
/// Controls the action of the variant_get kernel.
@@ -118,7 +309,7 @@ mod test {
use parquet_variant::{Variant, VariantPath};
use crate::json_to_variant;
- use crate::VariantArray;
+ use crate::{variant_array::ShreddedVariantFieldArray, VariantArray};
use super::{variant_get, GetOptions};
@@ -302,7 +493,6 @@ mod test {
}
/// Shredding: extract a value as an Int32Array, unsafe cast (should error on "n/a")
-
#[test]
fn get_variant_shredded_int32_as_int32_unsafe_cast() {
// Extract the typed value as Int32Array
@@ -762,4 +952,1043 @@ mod test {
VariantArray::try_new(Arc::new(struct_array)).expect("should create variant array"),
)
}
+ /// This test manually constructs a shredded variant array representing objects
+ /// like {"x": 1, "y": "foo"} and {"x": 42} and tests extracting the "x" field
+ /// as VariantArray using variant_get.
+ #[test]
+ fn test_shredded_object_field_access() {
+ let array = shredded_object_with_x_field_variant_array();
+
+ // Test: Extract the "x" field as VariantArray first
+ let options = GetOptions::new_with_path(VariantPath::from("x"));
+ let result = variant_get(&array, options).unwrap();
+
+ let result_variant: &VariantArray = result.as_any().downcast_ref().unwrap();
+ assert_eq!(result_variant.len(), 2);
+
+ // Row 0: expect x=1
+ assert_eq!(result_variant.value(0), Variant::Int32(1));
+ // Row 1: expect x=42
+ assert_eq!(result_variant.value(1), Variant::Int32(42));
+ }
+
+ /// Test extracting shredded object field with type conversion
+ #[test]
+ fn test_shredded_object_field_as_int32() {
+ let array = shredded_object_with_x_field_variant_array();
+
+ // Test: Extract the "x" field as Int32Array (type conversion)
+ let field = Field::new("x", DataType::Int32, false);
+ let options = GetOptions::new_with_path(VariantPath::from("x"))
+ .with_as_type(Some(FieldRef::from(field)));
+ let result = variant_get(&array, options).unwrap();
+
+ // Should get Int32Array
+ let expected: ArrayRef = Arc::new(Int32Array::from(vec![Some(1), Some(42)]));
+ assert_eq!(&result, &expected);
+ }
+
+ /// Helper function to create a shredded variant array representing objects
+ ///
+ /// This creates an array that represents:
+ /// Row 0: {"x": 1, "y": "foo"} (x is shredded, y is in value field)
+ /// Row 1: {"x": 42} (x is shredded, perfect shredding)
+ ///
+ /// The physical layout follows the shredding spec where:
+ /// - metadata: contains object metadata
+ /// - typed_value: StructArray with field "x" (ShreddedVariantFieldArray)
+ /// - value: contains fallback for unshredded fields like {"y": "foo"}
+ /// - The "x" field has typed_value=Int32Array and value=NULL (perfect shredding)
+ fn shredded_object_with_x_field_variant_array() -> ArrayRef {
+ // Create the base metadata for objects
+ let (metadata, y_field_value) = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let mut obj = builder.new_object();
+ obj.insert("x", Variant::Int32(42));
+ obj.insert("y", Variant::from("foo"));
+ obj.finish();
+ builder.finish()
+ };
+
+ // Create metadata array (same for both rows)
+ let metadata_array = BinaryViewArray::from_iter_values(std::iter::repeat_n(&metadata, 2));
+
+ // Create the main value field per the 3-step shredding spec:
+ // Step 2: If field not in shredding schema, check value field
+ // Row 0: {"y": "foo"} (y is not shredded, stays in value for step 2)
+ // Row 1: {} (empty object - no unshredded fields)
+ let empty_object_value = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let obj = builder.new_object();
+ obj.finish();
+ let (_, value) = builder.finish();
+ value
+ };
+
+ let value_array = BinaryViewArray::from(vec![
+ Some(y_field_value.as_slice()), // Row 0 has {"y": "foo"}
+ Some(empty_object_value.as_slice()), // Row 1 has {}
+ ]);
+
+ // Create the "x" field as a ShreddedVariantFieldArray
+ // This represents the shredded Int32 values for the "x" field
+ let x_field_typed_value = Int32Array::from(vec![Some(1), Some(42)]);
+
+ // For perfect shredding of the x field, no "value" column, only typed_value
+ let x_field_struct = crate::variant_array::StructArrayBuilder::new()
+ .with_field("typed_value", Arc::new(x_field_typed_value))
+ .build();
+
+ // Wrap the x field struct in a ShreddedVariantFieldArray
+ let x_field_shredded = ShreddedVariantFieldArray::try_new(Arc::new(x_field_struct))
+ .expect("should create ShreddedVariantFieldArray");
+
+ // Create the main typed_value as a struct containing the "x" field
+ let typed_value_fields = Fields::from(vec![Field::new(
+ "x",
+ x_field_shredded.data_type().clone(),
+ true,
+ )]);
+ let typed_value_struct = StructArray::try_new(
+ typed_value_fields,
+ vec![Arc::new(x_field_shredded)],
+ None, // No nulls - both rows have the object structure
+ )
+ .unwrap();
+
+ // Create the main VariantArray
+ let main_struct = crate::variant_array::StructArrayBuilder::new()
+ .with_field("metadata", Arc::new(metadata_array))
+ .with_field("value", Arc::new(value_array))
+ .with_field("typed_value", Arc::new(typed_value_struct))
+ .build();
+
+ Arc::new(VariantArray::try_new(Arc::new(main_struct)).expect("should create variant array"))
+ }
+
+ /// Simple test to check if nested paths are supported by current implementation
+ #[test]
+ fn test_simple_nested_path_support() {
+ // Check: How does VariantPath parse different strings?
+ println!("Testing path parsing:");
+
+ let path_x = VariantPath::from("x");
+ let elements_x: Vec<_> = path_x.iter().collect();
+ println!(" 'x' -> {} elements: {:?}", elements_x.len(), elements_x);
+
+ let path_ax = VariantPath::from("a.x");
+ let elements_ax: Vec<_> = path_ax.iter().collect();
+ println!(
+ " 'a.x' -> {} elements: {:?}",
+ elements_ax.len(),
+ elements_ax
+ );
+
+ let path_ax_alt = VariantPath::from("$.a.x");
+ let elements_ax_alt: Vec<_> = path_ax_alt.iter().collect();
+ println!(
+ " '$.a.x' -> {} elements: {:?}",
+ elements_ax_alt.len(),
+ elements_ax_alt
+ );
+
+ let path_nested = VariantPath::from("a").join("x");
+ let elements_nested: Vec<_> = path_nested.iter().collect();
+ println!(
+ " VariantPath::from('a').join('x') -> {} elements: {:?}",
+ elements_nested.len(),
+ elements_nested
+ );
+
+ // Use your existing simple test data but try "a.x" instead of "x"
+ let array = shredded_object_with_x_field_variant_array();
+
+ // Test if variant_get with REAL nested path throws not implemented error
+ let real_nested_path = VariantPath::from("a").join("x");
+ let options = GetOptions::new_with_path(real_nested_path);
+ let result = variant_get(&array, options);
+
+ match result {
+ Ok(_) => {
+ println!("Nested path 'a.x' works unexpectedly!");
+ }
+ Err(e) => {
+ println!("Nested path 'a.x' error: {}", e);
+ if e.to_string().contains("not yet implemented")
+ || e.to_string().contains("NotYetImplemented")
+ {
+ println!("This is expected - nested paths are not implemented");
+ return;
+ }
+ // Any other error is also expected for now
+ println!("This shows nested paths need implementation");
+ }
+ }
+ }
+
+ /// Test comprehensive variant_get scenarios with Int32 conversion
+ /// Test depth 0: Direct field access "x" with Int32 conversion
+ /// Covers shredded vs non-shredded VariantArrays for simple field access
+ #[test]
+ fn test_depth_0_int32_conversion() {
+ println!("=== Testing Depth 0: Direct field access ===");
+
+ // Non-shredded test data: [{"x": 42}, {"x": "foo"}, {"y": 10}]
+ let unshredded_array = create_depth_0_test_data();
+
+ let field = Field::new("result", DataType::Int32, true);
+ let path = VariantPath::from("x");
+ let options = GetOptions::new_with_path(path).with_as_type(Some(FieldRef::from(field)));
+ let result = variant_get(&unshredded_array, options).unwrap();
+
+ let expected: ArrayRef = Arc::new(Int32Array::from(vec![
+ Some(42), // {"x": 42} -> 42
+ None, // {"x": "foo"} -> NULL (type mismatch)
+ None, // {"y": 10} -> NULL (field missing)
+ ]));
+ assert_eq!(&result, &expected);
+ println!("Depth 0 (unshredded) passed");
+
+ // Shredded test data: using simplified approach based on working pattern
+ let shredded_array = create_depth_0_shredded_test_data_simple();
+
+ let field = Field::new("result", DataType::Int32, true);
+ let path = VariantPath::from("x");
+ let options = GetOptions::new_with_path(path).with_as_type(Some(FieldRef::from(field)));
+ let result = variant_get(&shredded_array, options).unwrap();
+
+ let expected: ArrayRef = Arc::new(Int32Array::from(vec![
+ Some(42), // {"x": 42} -> 42 (from typed_value)
+ None, // {"x": "foo"} -> NULL (type mismatch, from value field)
+ ]));
+ assert_eq!(&result, &expected);
+ println!("Depth 0 (shredded) passed");
+ }
+
+ /// Test depth 1: Single nested field access "a.x" with Int32 conversion
+ /// Covers shredded vs non-shredded VariantArrays for nested field access
+ #[test]
+ fn test_depth_1_int32_conversion() {
+ println!("=== Testing Depth 1: Single nested field access ===");
+
+ // Non-shredded test data from the GitHub issue
+ let unshredded_array = create_nested_path_test_data();
+
+ let field = Field::new("result", DataType::Int32, true);
+ let path = VariantPath::from("a.x"); // Dot notation!
+ let options = GetOptions::new_with_path(path).with_as_type(Some(FieldRef::from(field)));
+ let result = variant_get(&unshredded_array, options).unwrap();
+
+ let expected: ArrayRef = Arc::new(Int32Array::from(vec![
+ Some(55), // {"a": {"x": 55}} -> 55
+ None, // {"a": {"x": "foo"}} -> NULL (type mismatch)
+ ]));
+ assert_eq!(&result, &expected);
+ println!("Depth 1 (unshredded) passed");
+
+ // Shredded test data: depth 1 nested shredding
+ let shredded_array = create_depth_1_shredded_test_data_working();
+
+ let field = Field::new("result", DataType::Int32, true);
+ let path = VariantPath::from("a.x"); // Dot notation!
+ let options = GetOptions::new_with_path(path).with_as_type(Some(FieldRef::from(field)));
+ let result = variant_get(&shredded_array, options).unwrap();
+
+ let expected: ArrayRef = Arc::new(Int32Array::from(vec![
+ Some(55), // {"a": {"x": 55}} -> 55 (from nested shredded x)
+ None, // {"a": {"x": "foo"}} -> NULL (type mismatch in nested value)
+ ]));
+ assert_eq!(&result, &expected);
+ println!("Depth 1 (shredded) passed");
+ }
+
+ /// Test depth 2: Double nested field access "a.b.x" with Int32 conversion
+ /// Covers shredded vs non-shredded VariantArrays for deeply nested field access
+ #[test]
+ fn test_depth_2_int32_conversion() {
+ println!("=== Testing Depth 2: Double nested field access ===");
+
+ // Non-shredded test data: [{"a": {"b": {"x": 100}}}, {"a": {"b": {"x": "bar"}}}, {"a": {"b": {"y": 200}}}]
+ let unshredded_array = create_depth_2_test_data();
+
+ let field = Field::new("result", DataType::Int32, true);
+ let path = VariantPath::from("a.b.x"); // Double nested dot notation!
+ let options = GetOptions::new_with_path(path).with_as_type(Some(FieldRef::from(field)));
+ let result = variant_get(&unshredded_array, options).unwrap();
+
+ let expected: ArrayRef = Arc::new(Int32Array::from(vec![
+ Some(100), // {"a": {"b": {"x": 100}}} -> 100
+ None, // {"a": {"b": {"x": "bar"}}} -> NULL (type mismatch)
+ None, // {"a": {"b": {"y": 200}}} -> NULL (field missing)
+ ]));
+ assert_eq!(&result, &expected);
+ println!("Depth 2 (unshredded) passed");
+
+ // Shredded test data: depth 2 nested shredding
+ let shredded_array = create_depth_2_shredded_test_data_working();
+
+ let field = Field::new("result", DataType::Int32, true);
+ let path = VariantPath::from("a.b.x"); // Double nested dot notation!
+ let options = GetOptions::new_with_path(path).with_as_type(Some(FieldRef::from(field)));
+ let result = variant_get(&shredded_array, options).unwrap();
+
+ let expected: ArrayRef = Arc::new(Int32Array::from(vec![
+ Some(100), // {"a": {"b": {"x": 100}}} -> 100 (from deeply nested shredded x)
+ None, // {"a": {"b": {"x": "bar"}}} -> NULL (type mismatch in deep value)
+ None, // {"a": {"b": {"y": 200}}} -> NULL (field missing in deep structure)
+ ]));
+ assert_eq!(&result, &expected);
+ println!("Depth 2 (shredded) passed");
+ }
+
+ /// Test that demonstrates what CURRENTLY WORKS
+ ///
+ /// This shows that nested path functionality does work, but only when the
+ /// test data matches what the current implementation expects
+ #[test]
+ fn test_current_nested_path_functionality() {
+ let array = shredded_object_with_x_field_variant_array();
+
+ // Test: Extract the "x" field (single level) - this works
+ let single_path = VariantPath::from("x");
+ let field = Field::new("result", DataType::Int32, true);
+ let options =
+ GetOptions::new_with_path(single_path).with_as_type(Some(FieldRef::from(field)));
+ let result = variant_get(&array, options).unwrap();
+
+ println!("Single path 'x' works - result: {:?}", result);
+
+ // Test: Try nested path "a.x" - this is what we need to implement
+ let nested_path = VariantPath::from("a").join("x");
+ let field = Field::new("result", DataType::Int32, true);
+ let options =
+ GetOptions::new_with_path(nested_path).with_as_type(Some(FieldRef::from(field)));
+ let result = variant_get(&array, options).unwrap();
+
+ println!("Nested path 'a.x' result: {:?}", result);
+ }
+
+ /// Create test data for depth 0 (direct field access)
+ /// [{"x": 42}, {"x": "foo"}, {"y": 10}]
+ fn create_depth_0_test_data() -> ArrayRef {
+ let mut builder = crate::VariantArrayBuilder::new(3);
+
+ // Row 1: {"x": 42}
+ {
+ let json_str = r#"{"x": 42}"#;
+ let string_array: ArrayRef = Arc::new(StringArray::from(vec![json_str]));
+ if let Ok(variant_array) = json_to_variant(&string_array) {
+ builder.append_variant(variant_array.value(0));
+ } else {
+ builder.append_null();
+ }
+ }
+
+ // Row 2: {"x": "foo"}
+ {
+ let json_str = r#"{"x": "foo"}"#;
+ let string_array: ArrayRef = Arc::new(StringArray::from(vec![json_str]));
+ if let Ok(variant_array) = json_to_variant(&string_array) {
+ builder.append_variant(variant_array.value(0));
+ } else {
+ builder.append_null();
+ }
+ }
+
+ // Row 3: {"y": 10} (missing "x" field)
+ {
+ let json_str = r#"{"y": 10}"#;
+ let string_array: ArrayRef = Arc::new(StringArray::from(vec![json_str]));
+ if let Ok(variant_array) = json_to_variant(&string_array) {
+ builder.append_variant(variant_array.value(0));
+ } else {
+ builder.append_null();
+ }
+ }
+
+ Arc::new(builder.build())
+ }
+
+ /// Create test data for depth 1 (single nested field)
+ /// This represents the exact scenarios from the GitHub issue: "a.x"
+ fn create_nested_path_test_data() -> ArrayRef {
+ let mut builder = crate::VariantArrayBuilder::new(2);
+
+ // Row 1: {"a": {"x": 55}, "b": 42}
+ {
+ let json_str = r#"{"a": {"x": 55}, "b": 42}"#;
+ let string_array: ArrayRef = Arc::new(StringArray::from(vec![json_str]));
+ if let Ok(variant_array) = json_to_variant(&string_array) {
+ builder.append_variant(variant_array.value(0));
+ } else {
+ builder.append_null();
+ }
+ }
+
+ // Row 2: {"a": {"x": "foo"}, "b": 42}
+ {
+ let json_str = r#"{"a": {"x": "foo"}, "b": 42}"#;
+ let string_array: ArrayRef = Arc::new(StringArray::from(vec![json_str]));
+ if let Ok(variant_array) = json_to_variant(&string_array) {
+ builder.append_variant(variant_array.value(0));
+ } else {
+ builder.append_null();
+ }
+ }
+
+ Arc::new(builder.build())
+ }
+
+ /// Create test data for depth 2 (double nested field)
+ /// [{"a": {"b": {"x": 100}}}, {"a": {"b": {"x": "bar"}}}, {"a": {"b": {"y": 200}}}]
+ fn create_depth_2_test_data() -> ArrayRef {
+ let mut builder = crate::VariantArrayBuilder::new(3);
+
+ // Row 1: {"a": {"b": {"x": 100}}}
+ {
+ let json_str = r#"{"a": {"b": {"x": 100}}}"#;
+ let string_array: ArrayRef = Arc::new(StringArray::from(vec![json_str]));
+ if let Ok(variant_array) = json_to_variant(&string_array) {
+ builder.append_variant(variant_array.value(0));
+ } else {
+ builder.append_null();
+ }
+ }
+
+ // Row 2: {"a": {"b": {"x": "bar"}}}
+ {
+ let json_str = r#"{"a": {"b": {"x": "bar"}}}"#;
+ let string_array: ArrayRef = Arc::new(StringArray::from(vec![json_str]));
+ if let Ok(variant_array) = json_to_variant(&string_array) {
+ builder.append_variant(variant_array.value(0));
+ } else {
+ builder.append_null();
+ }
+ }
+
+ // Row 3: {"a": {"b": {"y": 200}}} (missing "x" field)
+ {
+ let json_str = r#"{"a": {"b": {"y": 200}}}"#;
+ let string_array: ArrayRef = Arc::new(StringArray::from(vec![json_str]));
+ if let Ok(variant_array) = json_to_variant(&string_array) {
+ builder.append_variant(variant_array.value(0));
+ } else {
+ builder.append_null();
+ }
+ }
+
+ Arc::new(builder.build())
+ }
+
+ /// Create simple shredded test data for depth 0 using a simplified working pattern
+ /// Creates 2 rows: [{"x": 42}, {"x": "foo"}] with "x" shredded where possible
+ fn create_depth_0_shredded_test_data_simple() -> ArrayRef {
+ // Create base metadata using the working pattern
+ let (metadata, string_x_value) = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let mut obj = builder.new_object();
+ obj.insert("x", Variant::from("foo"));
+ obj.finish();
+ builder.finish()
+ };
+
+ // Metadata array (same for both rows)
+ let metadata_array = BinaryViewArray::from_iter_values(std::iter::repeat_n(&metadata, 2));
+
+ // Value array following the 3-step shredding spec:
+ // Row 0: {} (x is shredded, no unshredded fields)
+ // Row 1: {"x": "foo"} (x is a string, can't be shredded to Int32)
+ let empty_object_value = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let obj = builder.new_object();
+ obj.finish();
+ let (_, value) = builder.finish();
+ value
+ };
+
+ let value_array = BinaryViewArray::from(vec![
+ Some(empty_object_value.as_slice()), // Row 0: {} (x shredded out)
+ Some(string_x_value.as_slice()), // Row 1: {"x": "foo"} (fallback)
+ ]);
+
+ // Create the "x" field as a ShreddedVariantFieldArray
+ let x_field_typed_value = Int32Array::from(vec![Some(42), None]);
+
+ // For the x field, only typed_value (perfect shredding when possible)
+ let x_field_struct = crate::variant_array::StructArrayBuilder::new()
+ .with_field("typed_value", Arc::new(x_field_typed_value))
+ .build();
+
+ let x_field_shredded = ShreddedVariantFieldArray::try_new(Arc::new(x_field_struct))
+ .expect("should create ShreddedVariantFieldArray");
+
+ // Create the main typed_value as a struct containing the "x" field
+ let typed_value_fields = Fields::from(vec![Field::new(
+ "x",
+ x_field_shredded.data_type().clone(),
+ true,
+ )]);
+ let typed_value_struct =
+ StructArray::try_new(typed_value_fields, vec![Arc::new(x_field_shredded)], None)
+ .unwrap();
+
+ // Build final VariantArray
+ let struct_array = crate::variant_array::StructArrayBuilder::new()
+ .with_field("metadata", Arc::new(metadata_array))
+ .with_field("value", Arc::new(value_array))
+ .with_field("typed_value", Arc::new(typed_value_struct))
+ .build();
+
+ Arc::new(VariantArray::try_new(Arc::new(struct_array)).expect("should create VariantArray"))
+ }
+
+ /// Create working depth 1 shredded test data based on the existing working pattern
+ /// This creates a properly structured shredded variant for "a.x" where:
+ /// - Row 0: {"a": {"x": 55}, "b": 42} with a.x shredded into typed_value
+ /// - Row 1: {"a": {"x": "foo"}, "b": 42} with a.x fallback to value field due to type mismatch
+ fn create_depth_1_shredded_test_data_working() -> ArrayRef {
+ // Create metadata following the working pattern from shredded_object_with_x_field_variant_array
+ let (metadata, _) = {
+ // Create nested structure: {"a": {"x": 55}, "b": 42}
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let mut obj = builder.new_object();
+
+ // Create the nested "a" object
+ let mut a_obj = obj.new_object("a");
+ a_obj.insert("x", Variant::Int32(55));
+ a_obj.finish();
+
+ obj.insert("b", Variant::Int32(42));
+ obj.finish();
+ builder.finish()
+ };
+
+ let metadata_array = BinaryViewArray::from_iter_values(std::iter::repeat_n(&metadata, 2));
+
+ // Create value arrays for the fallback case
+ // Following the spec: if field cannot be shredded, it stays in value
+ let empty_object_value = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let obj = builder.new_object();
+ obj.finish();
+ let (_, value) = builder.finish();
+ value
+ };
+
+ // Row 1 fallback: use the working pattern from the existing shredded test
+ // This avoids metadata issues by using the simple fallback approach
+ let row1_fallback = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let mut obj = builder.new_object();
+ obj.insert("fallback", Variant::from("data"));
+ obj.finish();
+ let (_, value) = builder.finish();
+ value
+ };
+
+ let value_array = BinaryViewArray::from(vec![
+ Some(empty_object_value.as_slice()), // Row 0: {} (everything shredded except b in unshredded fields)
+ Some(row1_fallback.as_slice()), // Row 1: {"a": {"x": "foo"}, "b": 42} (a.x can't be shredded)
+ ]);
+
+ // Create the nested shredded structure
+ // Level 2: x field (the deepest level)
+ let x_typed_value = Int32Array::from(vec![Some(55), None]);
+ let x_field_struct = crate::variant_array::StructArrayBuilder::new()
+ .with_field("typed_value", Arc::new(x_typed_value))
+ .build();
+ let x_field_shredded = ShreddedVariantFieldArray::try_new(Arc::new(x_field_struct))
+ .expect("should create ShreddedVariantFieldArray for x");
+
+ // Level 1: a field containing x field + value field for fallbacks
+ // The "a" field needs both typed_value (for shredded x) and value (for fallback cases)
+
+ // Create the value field for "a" (for cases where a.x can't be shredded)
+ let a_value_data = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let obj = builder.new_object();
+ obj.finish();
+ let (_, value) = builder.finish();
+ value
+ };
+ let a_value_array = BinaryViewArray::from(vec![
+ None, // Row 0: x is shredded, so no value fallback needed
+ Some(a_value_data.as_slice()), // Row 1: fallback for a.x="foo" (but logic will check typed_value first)
+ ]);
+
+ let a_inner_fields = Fields::from(vec![Field::new(
+ "x",
+ x_field_shredded.data_type().clone(),
+ true,
+ )]);
+ let a_inner_struct = crate::variant_array::StructArrayBuilder::new()
+ .with_field(
+ "typed_value",
+ Arc::new(
+ StructArray::try_new(a_inner_fields, vec![Arc::new(x_field_shredded)], None)
+ .unwrap(),
+ ),
+ )
+ .with_field("value", Arc::new(a_value_array))
+ .build();
+ let a_field_shredded = ShreddedVariantFieldArray::try_new(Arc::new(a_inner_struct))
+ .expect("should create ShreddedVariantFieldArray for a");
+
+ // Level 0: main typed_value struct containing a field
+ let typed_value_fields = Fields::from(vec![Field::new(
+ "a",
+ a_field_shredded.data_type().clone(),
+ true,
+ )]);
+ let typed_value_struct =
+ StructArray::try_new(typed_value_fields, vec![Arc::new(a_field_shredded)], None)
+ .unwrap();
+
+ // Build final VariantArray
+ let struct_array = crate::variant_array::StructArrayBuilder::new()
+ .with_field("metadata", Arc::new(metadata_array))
+ .with_field("value", Arc::new(value_array))
+ .with_field("typed_value", Arc::new(typed_value_struct))
+ .build();
+
+ Arc::new(VariantArray::try_new(Arc::new(struct_array)).expect("should create VariantArray"))
+ }
+
+ /// Create working depth 2 shredded test data for "a.b.x" paths
+ /// This creates a 3-level nested shredded structure where:
+ /// - Row 0: {"a": {"b": {"x": 100}}} with a.b.x shredded into typed_value
+ /// - Row 1: {"a": {"b": {"x": "bar"}}} with type mismatch fallback
+ /// - Row 2: {"a": {"b": {"y": 200}}} with missing field fallback
+ fn create_depth_2_shredded_test_data_working() -> ArrayRef {
+ // Create metadata following the working pattern
+ let (metadata, _) = {
+ // Create deeply nested structure: {"a": {"b": {"x": 100}}}
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let mut obj = builder.new_object();
+
+ // Create the nested "a.b" structure
+ let mut a_obj = obj.new_object("a");
+ let mut b_obj = a_obj.new_object("b");
+ b_obj.insert("x", Variant::Int32(100));
+ b_obj.finish();
+ a_obj.finish();
+
+ obj.finish();
+ builder.finish()
+ };
+
+ let metadata_array = BinaryViewArray::from_iter_values(std::iter::repeat_n(&metadata, 3));
+
+ // Create value arrays for fallback cases
+ let empty_object_value = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let obj = builder.new_object();
+ obj.finish();
+ let (_, value) = builder.finish();
+ value
+ };
+
+ // Simple fallback values - avoiding complex nested metadata
+ let value_array = BinaryViewArray::from(vec![
+ Some(empty_object_value.as_slice()), // Row 0: fully shredded
+ Some(empty_object_value.as_slice()), // Row 1: fallback (simplified)
+ Some(empty_object_value.as_slice()), // Row 2: fallback (simplified)
+ ]);
+
+ // Create the deeply nested shredded structure: a.b.x
+
+ // Level 3: x field (deepest level)
+ let x_typed_value = Int32Array::from(vec![Some(100), None, None]);
+ let x_field_struct = crate::variant_array::StructArrayBuilder::new()
+ .with_field("typed_value", Arc::new(x_typed_value))
+ .build();
+ let x_field_shredded = ShreddedVariantFieldArray::try_new(Arc::new(x_field_struct))
+ .expect("should create ShreddedVariantFieldArray for x");
+
+ // Level 2: b field containing x field + value field
+ let b_value_data = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let obj = builder.new_object();
+ obj.finish();
+ let (_, value) = builder.finish();
+ value
+ };
+ let b_value_array = BinaryViewArray::from(vec![
+ None, // Row 0: x is shredded
+ Some(b_value_data.as_slice()), // Row 1: fallback for b.x="bar"
+ Some(b_value_data.as_slice()), // Row 2: fallback for b.y=200
+ ]);
+
+ let b_inner_fields = Fields::from(vec![Field::new(
+ "x",
+ x_field_shredded.data_type().clone(),
+ true,
+ )]);
+ let b_inner_struct = crate::variant_array::StructArrayBuilder::new()
+ .with_field(
+ "typed_value",
+ Arc::new(
+ StructArray::try_new(b_inner_fields, vec![Arc::new(x_field_shredded)], None)
+ .unwrap(),
+ ),
+ )
+ .with_field("value", Arc::new(b_value_array))
+ .build();
+ let b_field_shredded = ShreddedVariantFieldArray::try_new(Arc::new(b_inner_struct))
+ .expect("should create ShreddedVariantFieldArray for b");
+
+ // Level 1: a field containing b field + value field
+ let a_value_data = {
+ let mut builder = parquet_variant::VariantBuilder::new();
+ let obj = builder.new_object();
+ obj.finish();
+ let (_, value) = builder.finish();
+ value
+ };
+ let a_value_array = BinaryViewArray::from(vec![
+ None, // Row 0: b is shredded
+ Some(a_value_data.as_slice()), // Row 1: fallback for a.b.*
+ Some(a_value_data.as_slice()), // Row 2: fallback for a.b.*
+ ]);
+
+ let a_inner_fields = Fields::from(vec![Field::new(
+ "b",
+ b_field_shredded.data_type().clone(),
+ true,
+ )]);
+ let a_inner_struct = crate::variant_array::StructArrayBuilder::new()
+ .with_field(
+ "typed_value",
+ Arc::new(
+ StructArray::try_new(a_inner_fields, vec![Arc::new(b_field_shredded)], None)
+ .unwrap(),
+ ),
+ )
+ .with_field("value", Arc::new(a_value_array))
+ .build();
+ let a_field_shredded = ShreddedVariantFieldArray::try_new(Arc::new(a_inner_struct))
+ .expect("should create ShreddedVariantFieldArray for a");
+
+ // Level 0: main typed_value struct containing a field
+ let typed_value_fields = Fields::from(vec![Field::new(
+ "a",
+ a_field_shredded.data_type().clone(),
+ true,
+ )]);
+ let typed_value_struct =
+ StructArray::try_new(typed_value_fields, vec![Arc::new(a_field_shredded)], None)
+ .unwrap();
+
+ // Build final VariantArray
+ let struct_array = crate::variant_array::StructArrayBuilder::new()
+ .with_field("metadata", Arc::new(metadata_array))
+ .with_field("value", Arc::new(value_array))
+ .with_field("typed_value", Arc::new(typed_value_struct))
+ .build();
+
+ Arc::new(VariantArray::try_new(Arc::new(struct_array)).expect("should create VariantArray"))
+ }
+
+ #[test]
+ fn test_strict_cast_options_downcast_failure() {
+ use arrow::compute::CastOptions;
+ use arrow::datatypes::{DataType, Field};
+ use arrow::error::ArrowError;
+ use parquet_variant::VariantPath;
+ use std::sync::Arc;
+
+ // Use the existing simple test data that has Int32 as typed_value
+ let variant_array = perfectly_shredded_int32_variant_array();
+
+ // Try to access a field with safe cast options (should return NULLs)
+ let safe_options = GetOptions {
+ path: VariantPath::from("nonexistent_field"),
+ as_type: Some(Arc::new(Field::new("result", DataType::Int32, true))),
+ cast_options: CastOptions::default(), // safe = true
+ };
+
+ let variant_array_ref: Arc<dyn arrow::array::Array> = variant_array.clone();
+ let result = variant_get(&variant_array_ref, safe_options);
+ // Should succeed and return NULLs (safe behavior)
+ assert!(result.is_ok());
+ let result_array = result.unwrap();
+ assert_eq!(result_array.len(), 3);
+ assert!(result_array.is_null(0));
+ assert!(result_array.is_null(1));
+ assert!(result_array.is_null(2));
+
+ // Try to access a field with strict cast options (should error)
+ let strict_options = GetOptions {
+ path: VariantPath::from("nonexistent_field"),
+ as_type: Some(Arc::new(Field::new("result", DataType::Int32, true))),
+ cast_options: CastOptions {
+ safe: false,
+ ..Default::default()
+ },
+ };
+
+ let result = variant_get(&variant_array_ref, strict_options);
+ // Should fail with a cast error
+ assert!(result.is_err());
+ let error = result.unwrap_err();
+ assert!(matches!(error, ArrowError::CastError(_)));
+ assert!(error
+ .to_string()
+ .contains("Cannot access field 'nonexistent_field' on non-struct type"));
+ }
+
+ #[test]
+ fn test_null_buffer_union_for_shredded_paths() {
+ use arrow::compute::CastOptions;
+ use arrow::datatypes::{DataType, Field};
+ use parquet_variant::VariantPath;
+ use std::sync::Arc;
+
+ // Test that null buffers are properly unioned when traversing shredded paths
+ // This test verifies scovich's null buffer union requirement
+
+ // Create a depth-1 shredded variant array where:
+ // - The top-level variant array has some nulls
+ // - The nested typed_value also has some nulls
+ // - The result should be the union of both null buffers
+
+ let variant_array = create_depth_1_shredded_test_data_working();
+
+ // Get the field "x" which should union nulls from:
+ // 1. The top-level variant array nulls
+ // 2. The "a" field's typed_value nulls
+ // 3. The "x" field's typed_value nulls
+ let options = GetOptions {
+ path: VariantPath::from("a.x"),
+ as_type: Some(Arc::new(Field::new("result", DataType::Int32, true))),
+ cast_options: CastOptions::default(),
+ };
+
+ let variant_array_ref: Arc<dyn arrow::array::Array> = variant_array.clone();
+ let result = variant_get(&variant_array_ref, options).unwrap();
+
+ // Verify the result length matches input
+ assert_eq!(result.len(), variant_array.len());
+
+ // The null pattern should reflect the union of all ancestor nulls
+ // Row 0: Should have valid data (path exists and is shredded as Int32)
+ // Row 1: Should be null (due to type mismatch - "foo" can't cast to Int32)
+ assert!(!result.is_null(0), "Row 0 should have valid Int32 data");
+ assert!(
+ result.is_null(1),
+ "Row 1 should be null due to type casting failure"
+ );
+
+ // Verify the actual values
+ let int32_result = result
+ .as_any()
+ .downcast_ref::<arrow::array::Int32Array>()
+ .unwrap();
+ assert_eq!(int32_result.value(0), 55); // The valid Int32 value
+ }
+
+ #[test]
+ fn test_struct_null_mask_union_from_children() {
+ use arrow::compute::CastOptions;
+ use arrow::datatypes::{DataType, Field, Fields};
+ use parquet_variant::VariantPath;
+ use std::sync::Arc;
+
+ use arrow::array::StringArray;
+
+ // Test that struct null masks properly union nulls from children field extractions
+ // This verifies scovich's concern about incomplete null masks in struct construction
+
+ // Create test data where some fields will fail type casting
+ let json_strings = vec![
+ r#"{"a": 42, "b": "hello"}"#, // Row 0: a=42 (castable to int), b="hello" (not castable to int)
+ r#"{"a": "world", "b": 100}"#, // Row 1: a="world" (not castable to int), b=100 (castable to int)
+ r#"{"a": 55, "b": 77}"#, // Row 2: a=55 (castable to int), b=77 (castable to int)
+ ];
+
+ let string_array: Arc<dyn arrow::array::Array> = Arc::new(StringArray::from(json_strings));
+ let variant_array = json_to_variant(&string_array).unwrap();
+
+ // Request extraction as a struct with both fields as Int32
+ // This should create child arrays where some fields are null due to casting failures
+ let struct_fields = Fields::from(vec![
+ Field::new("a", DataType::Int32, true),
+ Field::new("b", DataType::Int32, true),
+ ]);
+ let struct_type = DataType::Struct(struct_fields);
+
+ let options = GetOptions {
+ path: VariantPath::default(), // Extract the whole object as struct
+ as_type: Some(Arc::new(Field::new("result", struct_type, true))),
+ cast_options: CastOptions::default(),
+ };
+
+ let variant_array_ref: Arc<dyn arrow::array::Array> = Arc::new(variant_array);
+ let result = variant_get(&variant_array_ref, options).unwrap();
+
+ // Verify the result is a StructArray
+ let struct_result = result
+ .as_any()
+ .downcast_ref::<arrow::array::StructArray>()
+ .unwrap();
+ assert_eq!(struct_result.len(), 3);
+
+ // Get the individual field arrays
+ let field_a = struct_result
+ .column(0)
+ .as_any()
+ .downcast_ref::<arrow::array::Int32Array>()
+ .unwrap();
+ let field_b = struct_result
+ .column(1)
+ .as_any()
+ .downcast_ref::<arrow::array::Int32Array>()
+ .unwrap();
+
+ // Verify field values and nulls
+ // Row 0: a=42 (valid), b=null (casting failure)
+ assert!(!field_a.is_null(0));
+ assert_eq!(field_a.value(0), 42);
+ assert!(field_b.is_null(0)); // "hello" can't cast to int
+
+ // Row 1: a=null (casting failure), b=100 (valid)
+ assert!(field_a.is_null(1)); // "world" can't cast to int
+ assert!(!field_b.is_null(1));
+ assert_eq!(field_b.value(1), 100);
+
+ // Row 2: a=55 (valid), b=77 (valid)
+ assert!(!field_a.is_null(2));
+ assert_eq!(field_a.value(2), 55);
+ assert!(!field_b.is_null(2));
+ assert_eq!(field_b.value(2), 77);
+
+ // Verify the struct-level null mask properly unions child nulls
+ // The struct should NOT be null in any row because each row has at least one valid field
+ // (This tests that we're not incorrectly making the entire struct null when children fail)
+ assert!(!struct_result.is_null(0)); // Has valid field 'a'
+ assert!(!struct_result.is_null(1)); // Has valid field 'b'
+ assert!(!struct_result.is_null(2)); // Has both valid fields
+ }
+
+ #[test]
+ fn test_field_nullability_preservation() {
+ use arrow::compute::CastOptions;
+ use arrow::datatypes::{DataType, Field};
+ use parquet_variant::VariantPath;
+ use std::sync::Arc;
+
+ use arrow::array::StringArray;
+
+ // Test that field nullability from GetOptions.as_type is preserved in the result
+
+ let json_strings = vec![
+ r#"{"x": 42}"#, // Row 0: Valid int that should convert to Int32
+ r#"{"x": "not_a_number"}"#, // Row 1: String that can't cast to Int32
+ r#"{"x": null}"#, // Row 2: Explicit null value
+ r#"{"x": "hello"}"#, // Row 3: Another string (wrong type)
+ r#"{"y": 100}"#, // Row 4: Missing "x" field (SQL NULL case)
+ r#"{"x": 127}"#, // Row 5: Small int (could be Int8, widening cast candidate)
+ r#"{"x": 32767}"#, // Row 6: Medium int (could be Int16, widening cast candidate)
+ r#"{"x": 2147483647}"#, // Row 7: Max Int32 value (fits in Int32)
+ r#"{"x": 9223372036854775807}"#, // Row 8: Large Int64 value (cannot convert to Int32)
+ ];
+
+ let string_array: Arc<dyn arrow::array::Array> = Arc::new(StringArray::from(json_strings));
+ let variant_array = json_to_variant(&string_array).unwrap();
+
+ // Test 1: nullable field (should allow nulls from cast failures)
+ let nullable_field = Arc::new(Field::new("result", DataType::Int32, true));
+ let options_nullable = GetOptions {
+ path: VariantPath::from("x"),
+ as_type: Some(nullable_field.clone()),
+ cast_options: CastOptions::default(),
+ };
+
+ let variant_array_ref: Arc<dyn arrow::array::Array> = Arc::new(variant_array);
+ let result_nullable = variant_get(&variant_array_ref, options_nullable).unwrap();
+
+ // Verify we get an Int32Array with nulls for cast failures
+ let int32_result = result_nullable
+ .as_any()
+ .downcast_ref::<arrow::array::Int32Array>()
+ .unwrap();
+ assert_eq!(int32_result.len(), 9);
+
+ // Row 0: 42 converts successfully to Int32
+ assert!(!int32_result.is_null(0));
+ assert_eq!(int32_result.value(0), 42);
+
+ // Row 1: "not_a_number" fails to convert -> NULL
+ assert!(int32_result.is_null(1));
+
+ // Row 2: explicit null value -> NULL
+ assert!(int32_result.is_null(2));
+
+ // Row 3: "hello" (wrong type) fails to convert -> NULL
+ assert!(int32_result.is_null(3));
+
+ // Row 4: missing "x" field (SQL NULL case) -> NULL
+ assert!(int32_result.is_null(4));
+
+ // Row 5: 127 (small int, potential Int8 -> Int32 widening)
+ // Current behavior: JSON parses to Int8, should convert to Int32
+ assert!(!int32_result.is_null(5));
+ assert_eq!(int32_result.value(5), 127);
+
+ // Row 6: 32767 (medium int, potential Int16 -> Int32 widening)
+ // Current behavior: JSON parses to Int16, should convert to Int32
+ assert!(!int32_result.is_null(6));
+ assert_eq!(int32_result.value(6), 32767);
+
+ // Row 7: 2147483647 (max Int32, fits exactly)
+ // Current behavior: Should convert successfully
+ assert!(!int32_result.is_null(7));
+ assert_eq!(int32_result.value(7), 2147483647);
+
+ // Row 8: 9223372036854775807 (large Int64, cannot fit in Int32)
+ // Current behavior: Should fail conversion -> NULL
+ assert!(int32_result.is_null(8));
+
+ // Test 2: non-nullable field (behavior should be the same with safe casting)
+ let non_nullable_field = Arc::new(Field::new("result", DataType::Int32, false));
+ let options_non_nullable = GetOptions {
+ path: VariantPath::from("x"),
+ as_type: Some(non_nullable_field.clone()),
+ cast_options: CastOptions::default(), // safe=true by default
+ };
+
+ // Create variant array again since we moved it
+ let variant_array_2 = json_to_variant(&string_array).unwrap();
+ let variant_array_ref_2: Arc<dyn arrow::array::Array> = Arc::new(variant_array_2);
+ let result_non_nullable = variant_get(&variant_array_ref_2, options_non_nullable).unwrap();
+ let int32_result_2 = result_non_nullable
+ .as_any()
+ .downcast_ref::<arrow::array::Int32Array>()
+ .unwrap();
+
+ // Even with a non-nullable field, safe casting should still produce nulls for failures
+ assert_eq!(int32_result_2.len(), 9);
+
+ // Row 0: 42 converts successfully to Int32
+ assert!(!int32_result_2.is_null(0));
+ assert_eq!(int32_result_2.value(0), 42);
+
+ // Rows 1-4: All should be null due to safe casting behavior
+ // (non-nullable field specification doesn't override safe casting behavior)
+ assert!(int32_result_2.is_null(1)); // "not_a_number"
+ assert!(int32_result_2.is_null(2)); // explicit null
+ assert!(int32_result_2.is_null(3)); // "hello"
+ assert!(int32_result_2.is_null(4)); // missing field
+
+ // Rows 5-7: These should also convert successfully (numeric widening/fitting)
+ assert!(!int32_result_2.is_null(5)); // 127 (Int8 -> Int32)
+ assert_eq!(int32_result_2.value(5), 127);
+ assert!(!int32_result_2.is_null(6)); // 32767 (Int16 -> Int32)
+ assert_eq!(int32_result_2.value(6), 32767);
+ assert!(!int32_result_2.is_null(7)); // 2147483647 (fits in Int32)
+ assert_eq!(int32_result_2.value(7), 2147483647);
+
+ // Row 8: Large Int64 should fail conversion -> NULL
+ assert!(int32_result_2.is_null(8)); // 9223372036854775807 (too large for Int32)
+ }
}
diff --git a/parquet-variant-compute/src/variant_get/output/mod.rs b/parquet-variant-compute/src/variant_get/output/mod.rs
index 3ca21d482f..c3df183ec8 100644
--- a/parquet-variant-compute/src/variant_get/output/mod.rs
+++ b/parquet-variant-compute/src/variant_get/output/mod.rs
@@ -15,85 +15,4 @@
// specific language governing permissions and limitations
// under the License.
-mod primitive;
-mod variant;
-
-use crate::variant_get::output::primitive::PrimitiveOutputBuilder;
-use crate::variant_get::output::variant::VariantOutputBuilder;
-use crate::variant_get::GetOptions;
-use crate::VariantArray;
-use arrow::array::{ArrayRef, BinaryViewArray};
-use arrow::datatypes::{Int16Type, Int32Type};
-use arrow::error::Result;
-use arrow_schema::{ArrowError, DataType};
-
-/// This trait represents something that gets the output of the variant_get kernel.
-///
-/// For example, there are specializations for writing the output as a VariantArray,
-/// or as a specific type (e.g. Int32Array).
-///
-/// See [`instantiate_output_builder`] to create an instance of this trait.
-pub(crate) trait OutputBuilder {
- /// create output for a shredded variant array
- fn partially_shredded(
- &self,
- variant_array: &VariantArray,
- metadata: &BinaryViewArray,
- value_field: &BinaryViewArray,
- typed_value: &ArrayRef,
- ) -> Result<ArrayRef>;
-
- /// output for a perfectly shredded variant array
- fn typed(
- &self,
- variant_array: &VariantArray,
- metadata: &BinaryViewArray,
- typed_value: &ArrayRef,
- ) -> Result<ArrayRef>;
-
- /// write out an unshredded variant array
- fn unshredded(
- &self,
- variant_array: &VariantArray,
- metadata: &BinaryViewArray,
- value_field: &BinaryViewArray,
- ) -> Result<ArrayRef>;
-
- /// write out an all-null variant array
- fn all_null(
- &self,
- variant_array: &VariantArray,
- metadata: &BinaryViewArray,
- ) -> Result<ArrayRef>;
-}
-
-pub(crate) fn instantiate_output_builder<'a>(
- options: GetOptions<'a>,
-) -> Result<Box<dyn OutputBuilder + 'a>> {
- let GetOptions {
- as_type,
- path,
- cast_options,
- } = options;
-
- let Some(as_type) = as_type else {
- return Ok(Box::new(VariantOutputBuilder::new(path)));
- };
-
- // handle typed output
- match as_type.data_type() {
- DataType::Int32 => Ok(Box::new(PrimitiveOutputBuilder::<Int32Type>::new(
- path,
- as_type,
- cast_options,
- ))),
- DataType::Int16 => Ok(Box::new(PrimitiveOutputBuilder::<Int16Type>::new(
- path,
- as_type,
- cast_options,
- ))),
- dt => Err(ArrowError::NotYetImplemented(format!(
- "variant_get with as_type={dt} is not implemented yet",
- ))),
- }
-}
+pub(crate) mod row_builder;
diff --git a/parquet-variant-compute/src/variant_get/output/row_builder.rs b/parquet-variant-compute/src/variant_get/output/row_builder.rs
new file mode 100644
index 0000000000..787bdd610d
--- /dev/null
+++ b/parquet-variant-compute/src/variant_get/output/row_builder.rs
@@ -0,0 +1,342 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::array::ArrayRef;
+use arrow::compute::CastOptions;
+use arrow::datatypes;
+use arrow::datatypes::ArrowPrimitiveType;
+use arrow::error::{ArrowError, Result};
+use parquet_variant::{Variant, VariantPath};
+
+use crate::VariantArrayBuilder;
+
+use std::sync::Arc;
+
+pub(crate) fn make_shredding_row_builder<'a>(
+ //metadata: &BinaryViewArray,
+ path: VariantPath<'a>,
+ data_type: Option<&'a datatypes::DataType>,
+ cast_options: &'a CastOptions,
+) -> Result<Box<dyn VariantShreddingRowBuilder + 'a>> {
+ use arrow::array::PrimitiveBuilder;
+ use datatypes::{
+ Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type,
+ };
+
+ // support non-empty paths (field access) and some empty path cases
+ if path.is_empty() {
+ return match data_type {
+ Some(datatypes::DataType::Int8) => {
+ let builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Int8Type>::new(),
+ cast_options,
+ };
+ Ok(Box::new(builder))
+ }
+ Some(datatypes::DataType::Int16) => {
+ let builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Int16Type>::new(),
+ cast_options,
+ };
+ Ok(Box::new(builder))
+ }
+ Some(datatypes::DataType::Int32) => {
+ let builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Int32Type>::new(),
+ cast_options,
+ };
+ Ok(Box::new(builder))
+ }
+ Some(datatypes::DataType::Int64) => {
+ let builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Int64Type>::new(),
+ cast_options,
+ };
+ Ok(Box::new(builder))
+ }
+ Some(datatypes::DataType::Float16) => {
+ let builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Float16Type>::new(),
+ cast_options,
+ };
+ Ok(Box::new(builder))
+ }
+ Some(datatypes::DataType::Float32) => {
+ let builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Float32Type>::new(),
+ cast_options,
+ };
+ Ok(Box::new(builder))
+ }
+ Some(datatypes::DataType::Float64) => {
+ let builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Float64Type>::new(),
+ cast_options,
+ };
+ Ok(Box::new(builder))
+ }
+ None => {
+ // Return VariantArrayBuilder for VariantArray output
+ let builder = VariantArrayShreddingRowBuilder::new(16);
+ Ok(Box::new(builder))
+ }
+ _ => Err(ArrowError::NotYetImplemented(format!(
+ "variant_get with empty path and data_type={:?} not yet implemented",
+ data_type
+ ))),
+ };
+ }
+
+ // Non-empty paths: field access functionality
+ // Helper macro to reduce duplication when wrapping builders with path functionality
+ macro_rules! wrap_with_path {
+ ($inner_builder:expr) => {
+ Ok(Box::new(VariantPathRowBuilder {
+ builder: $inner_builder,
+ path,
+ }) as Box<dyn VariantShreddingRowBuilder + 'a>)
+ };
+ }
+
+ match data_type {
+ Some(datatypes::DataType::Int8) => {
+ let inner_builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Int8Type>::new(),
+ cast_options,
+ };
+ wrap_with_path!(inner_builder)
+ }
+ Some(datatypes::DataType::Int16) => {
+ let inner_builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Int16Type>::new(),
+ cast_options,
+ };
+ wrap_with_path!(inner_builder)
+ }
+ Some(datatypes::DataType::Int32) => {
+ let inner_builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Int32Type>::new(),
+ cast_options,
+ };
+ wrap_with_path!(inner_builder)
+ }
+ Some(datatypes::DataType::Int64) => {
+ let inner_builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Int64Type>::new(),
+ cast_options,
+ };
+ wrap_with_path!(inner_builder)
+ }
+ Some(datatypes::DataType::Float16) => {
+ let inner_builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Float16Type>::new(),
+ cast_options,
+ };
+ wrap_with_path!(inner_builder)
+ }
+ Some(datatypes::DataType::Float32) => {
+ let inner_builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Float32Type>::new(),
+ cast_options,
+ };
+ wrap_with_path!(inner_builder)
+ }
+ Some(datatypes::DataType::Float64) => {
+ let inner_builder = PrimitiveVariantShreddingRowBuilder {
+ builder: PrimitiveBuilder::<Float64Type>::new(),
+ cast_options,
+ };
+ wrap_with_path!(inner_builder)
+ }
+ None => {
+ // Create a variant array builder and wrap it with path functionality
+ let inner_builder = VariantArrayShreddingRowBuilder::new(16);
+ wrap_with_path!(inner_builder)
+ }
+ _ => Err(ArrowError::NotYetImplemented(format!(
+ "variant_get with path={:?} and data_type={:?} not yet implemented",
+ path, data_type
+ ))),
+ }
+}
+
+/// Builder for shredding variant values into strongly typed Arrow arrays.
+///
+/// Useful for variant_get kernels that need to extract specific paths from variant values, possibly
+/// with casting of leaf values to specific types.
+pub(crate) trait VariantShreddingRowBuilder {
+ fn append_null(&mut self) -> Result<()>;
+
+ fn append_value(&mut self, value: &Variant<'_, '_>) -> Result<bool>;
+
+ fn finish(&mut self) -> Result<ArrayRef>;
+}
+
+/// A thin wrapper whose only job is to extract a specific path from a variant value and pass the
+/// result to a nested builder.
+struct VariantPathRowBuilder<'a, T: VariantShreddingRowBuilder> {
+ builder: T,
+ path: VariantPath<'a>,
+}
+
+impl<T: VariantShreddingRowBuilder> VariantShreddingRowBuilder for VariantPathRowBuilder<'_, T> {
+ fn append_null(&mut self) -> Result<()> {
+ self.builder.append_null()
+ }
+
+ fn append_value(&mut self, value: &Variant<'_, '_>) -> Result<bool> {
+ if let Some(v) = value.get_path(&self.path) {
+ self.builder.append_value(&v)
+ } else {
+ self.builder.append_null()?;
+ Ok(false)
+ }
+ }
+ fn finish(&mut self) -> Result<ArrayRef> {
+ self.builder.finish()
+ }
+}
+
+/// Helper trait for converting `Variant` values to arrow primitive values.
+trait VariantAsPrimitive<T: ArrowPrimitiveType> {
+ fn as_primitive(&self) -> Option<T::Native>;
+}
+
+impl VariantAsPrimitive<datatypes::Int32Type> for Variant<'_, '_> {
+ fn as_primitive(&self) -> Option<i32> {
+ self.as_int32()
+ }
+}
+impl VariantAsPrimitive<datatypes::Int16Type> for Variant<'_, '_> {
+ fn as_primitive(&self) -> Option<i16> {
+ self.as_int16()
+ }
+}
+impl VariantAsPrimitive<datatypes::Int8Type> for Variant<'_, '_> {
+ fn as_primitive(&self) -> Option<i8> {
+ self.as_int8()
+ }
+}
+impl VariantAsPrimitive<datatypes::Int64Type> for Variant<'_, '_> {
+ fn as_primitive(&self) -> Option<i64> {
+ self.as_int64()
+ }
+}
+impl VariantAsPrimitive<datatypes::Float16Type> for Variant<'_, '_> {
+ fn as_primitive(&self) -> Option<half::f16> {
+ self.as_f16()
+ }
+}
+impl VariantAsPrimitive<datatypes::Float32Type> for Variant<'_, '_> {
+ fn as_primitive(&self) -> Option<f32> {
+ self.as_f32()
+ }
+}
+impl VariantAsPrimitive<datatypes::Float64Type> for Variant<'_, '_> {
+ fn as_primitive(&self) -> Option<f64> {
+ self.as_f64()
+ }
+}
+
+/// Helper function to get a user-friendly type name
+fn get_type_name<T: ArrowPrimitiveType>() -> &'static str {
+ match std::any::type_name::<T>() {
+ "arrow_array::types::Int32Type" => "Int32",
+ "arrow_array::types::Int16Type" => "Int16",
+ "arrow_array::types::Int8Type" => "Int8",
+ "arrow_array::types::Int64Type" => "Int64",
+ "arrow_array::types::UInt32Type" => "UInt32",
+ "arrow_array::types::UInt16Type" => "UInt16",
+ "arrow_array::types::UInt8Type" => "UInt8",
+ "arrow_array::types::UInt64Type" => "UInt64",
+ "arrow_array::types::Float32Type" => "Float32",
+ "arrow_array::types::Float64Type" => "Float64",
+ "arrow_array::types::Float16Type" => "Float16",
+ _ => "Unknown",
+ }
+}
+
+/// Builder for shredding variant values to primitive values
+struct PrimitiveVariantShreddingRowBuilder<'a, T: ArrowPrimitiveType> {
+ builder: arrow::array::PrimitiveBuilder<T>,
+ cast_options: &'a CastOptions<'a>,
+}
+
+impl<'a, T> VariantShreddingRowBuilder for PrimitiveVariantShreddingRowBuilder<'a, T>
+where
+ T: ArrowPrimitiveType,
+ for<'m, 'v> Variant<'m, 'v>: VariantAsPrimitive<T>,
+{
+ fn append_null(&mut self) -> Result<()> {
+ self.builder.append_null();
+ Ok(())
+ }
+
+ fn append_value(&mut self, value: &Variant<'_, '_>) -> Result<bool> {
+ if let Some(v) = value.as_primitive() {
+ self.builder.append_value(v);
+ Ok(true)
+ } else {
+ if !self.cast_options.safe {
+ // Unsafe casting: return error on conversion failure
+ return Err(ArrowError::CastError(format!(
+ "Failed to extract primitive of type {} from variant {:?} at path VariantPath([])",
+ get_type_name::<T>(),
+ value
+ )));
+ }
+ // Safe casting: append null on conversion failure
+ self.builder.append_null();
+ Ok(false)
+ }
+ }
+
+ fn finish(&mut self) -> Result<ArrayRef> {
+ Ok(Arc::new(self.builder.finish()))
+ }
+}
+
+/// Builder for creating VariantArray output (for path extraction without type conversion)
+struct VariantArrayShreddingRowBuilder {
+ builder: VariantArrayBuilder,
+}
+
+impl VariantArrayShreddingRowBuilder {
+ fn new(capacity: usize) -> Self {
+ Self {
+ builder: VariantArrayBuilder::new(capacity),
+ }
+ }
+}
+
+impl VariantShreddingRowBuilder for VariantArrayShreddingRowBuilder {
+ fn append_null(&mut self) -> Result<()> {
+ self.builder.append_null();
+ Ok(())
+ }
+
+ fn append_value(&mut self, value: &Variant<'_, '_>) -> Result<bool> {
+ self.builder.append_variant(value.clone());
+ Ok(true)
+ }
+
+ fn finish(&mut self) -> Result<ArrayRef> {
+ // VariantArrayBuilder::build takes ownership, so we need to replace it
+ let builder = std::mem::replace(&mut self.builder, VariantArrayBuilder::new(0));
+ Ok(Arc::new(builder.build()))
+ }
+}
diff --git a/parquet-variant/src/path.rs b/parquet-variant/src/path.rs
index 3ba50da328..794636ef40 100644
--- a/parquet-variant/src/path.rs
+++ b/parquet-variant/src/path.rs
@@ -95,10 +95,10 @@ impl<'a> From<Vec<VariantPathElement<'a>>> for VariantPath<'a> {
}
}
-/// Create from &str
+/// Create from &str with support for dot notation
impl<'a> From<&'a str> for VariantPath<'a> {
fn from(path: &'a str) -> Self {
- VariantPath::new(vec![path.into()])
+ VariantPath::new(path.split('.').map(Into::into).collect())
}
}
@@ -109,6 +109,12 @@ impl<'a> From<usize> for VariantPath<'a> {
}
}
+impl<'a> From<&[VariantPathElement<'a>]> for VariantPath<'a> {
+ fn from(elements: &[VariantPathElement<'a>]) -> Self {
+ VariantPath::new(elements.to_vec())
+ }
+}
+
/// Create from iter
impl<'a> FromIterator<VariantPathElement<'a>> for VariantPath<'a> {
fn from_iter<T: IntoIterator<Item = VariantPathElement<'a>>>(iter: T) -> Self {