You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/07/29 21:28:01 UTC
[PR] WIP [Variant] Strawman `variant_get` for shredded variants [arrow-rs]
alamb opened a new pull request, #8021:
URL: https://github.com/apache/arrow-rs/pull/8021
(this PR is not ready for review -- it doesn't actually implement variant get, it just implements the scaffolding for typed variants in VariantArray and the `variant_get` kernel)
# Which issue does this PR close?
- Part of https://github.com/apache/arrow-rs/issues/6736
- Part of https://github.com/apache/arrow-rs/issues/7941
- Closes https://github.com/apache/arrow-rs/pull/7965
# Rationale for this change
This is also based on some of the ideas from @carpecodeum in https://github.com/apache/arrow-rs/pull/7946 and @scovich in https://github.com/apache/arrow-rs/pull/7915
I basically took the tests from https://github.com/apache/arrow-rs/pull/7965 and the conversation with @scovich recorded from https://github.com/apache/arrow-rs/issues/7941#issuecomment-3090534233 and I bashed out how this might look
# What changes are included in this PR?
TODO
# Are these changes tested?
Yes
# Are there any user-facing changes?
New feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3155232515
> > > @scovich @carpecodeum and @Samyak2 this PR I think is ready for review
> >
> >
> > Whoop, I somehow missed the notify. I'll try to take a look in the next day or two!
>
> No worries -- I'll address @carpecodeum and @Samyak2 's comments in the mean time
Ping or request a review when ready?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2256898555
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
Review Comment:
Makes sense -- I'll file a follow on ticket to add support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2260892295
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -186,13 +340,11 @@ impl Array for VariantArray {
}
fn slice(&self, offset: usize, length: usize) -> ArrayRef {
- let slice = self.inner.slice(offset, length);
- let met = self.metadata_ref.slice(offset, length);
- let val = self.value_ref.slice(offset, length);
+ let inner = self.inner.slice(offset, length);
+ let shredding_state = self.shredding_state.slice(offset, length);
Arc::new(Self {
Review Comment:
I agree I don't expect any measurable performance improvement
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
+ let variant = variant_array.value(i);
+ let Some(value) = T::from_variant(&variant) else {
+ if self.cast_options.safe {
+ // safe mode: append null if we can't convert
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ } else {
+ return Err(ArrowError::CastError(format!(
+ "Failed to extract primitive of type {} from variant {:?} at path {:?}",
+ self.as_type.data_type(),
+ variant,
+ self.path
+ )));
+ }
+ };
+
+ nulls.append_non_null();
+ values.push(value)
+ } else {
+ // otherwise we have a typed value, so we can use it directly
+ nulls.append_non_null();
+ values.push(typed_value.value(i));
+ }
+ }
+
+ let nulls = nulls.finish();
+ let array = PrimitiveArray::<T>::new(values.into(), nulls)
+ .with_data_type(self.as_type.data_type().clone());
+ Ok(Arc::new(array))
+ }
+
+ fn fully_shredded(
+ &self,
+ _variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // if the types match exactly, we can just return the typed_value
+ if typed_value.data_type() == self.as_type.data_type() {
+ Ok(typed_value.clone())
+ } else {
+ // TODO: try to cast the typed_value to the desired type?
+ Err(ArrowError::NotYetImplemented(format!(
Review Comment:
Yeah, I am not sure what to do with this casting -- I will file a follow on ticket for us to sort it out
##########
parquet-variant-compute/src/variant_get/mod.rs:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow::{
+ array::{Array, ArrayRef},
+ compute::CastOptions,
+ error::Result,
+};
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::VariantPath;
+
+use crate::variant_array::ShreddingState;
+use crate::variant_get::output::instantiate_output_builder;
+use crate::VariantArray;
+
+mod output;
+
+/// Returns an array with the specified path extracted from the variant values.
+///
+/// The return array type depends on the `as_type` field of the options parameter
+/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
+/// to the specified path.
+/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
+ let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
+ ArrowError::InvalidArgumentError(
+ "expected a VariantArray as the input for variant_get".to_owned(),
+ )
+ })?;
+
+ // Create the output writer based on the specified output options
+ let output_builder = instantiate_output_builder(options.clone())?;
+
+ // Dispatch based on the shredding state of the input variant array
+ match variant_array.shredding_state() {
Review Comment:
Yeah, exactly, this I think is the key challenge of this code -- each different type of output needs to handle all the possible input shredding types as well.
##########
parquet-variant-compute/src/variant_get/output/mod.rs:
##########
Review Comment:
I think this aligns with my thinking and is nicely described -- I will use it as the base for the next set of tickets perhaps
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
> We'd either have to do all the pathing here (recursively), or caller would have to to extract the struct/array once and recurse on its fields.
Yes, I was imagining that this would do the pathing here on each row -- I am not sure what you mean by extract the struct array (as I don't think we can extract variants as structs in general, as the structures vary row by row)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2256914173
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ ))),
+ }
+ }
+
+ /// Return a reference to the metadata field
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ match self {
+ ShreddingState::Unshredded { metadata, .. } => metadata,
+ ShreddingState::FullyShredded { metadata, .. } => metadata,
+ ShreddingState::PartiallyShredded { metadata, .. } => metadata,
+ }
+ }
+
+ /// Return a reference to the value field, if present
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ match self {
+ ShreddingState::Unshredded { value, .. } => Some(value),
+ ShreddingState::FullyShredded { .. } => None,
+ ShreddingState::PartiallyShredded { value, .. } => Some(value),
+ }
+ }
+
+ /// Return a reference to the typed_value field, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ match self {
+ ShreddingState::Unshredded { .. } => None,
+ ShreddingState::FullyShredded { typed_value, .. } => Some(typed_value),
+ ShreddingState::PartiallyShredded { typed_value, .. } => Some(typed_value),
+ }
+ }
+
+ /// Slice all the underlying arrays
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ match self {
+ ShreddingState::Unshredded { metadata, value } => ShreddingState::Unshredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ },
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => ShreddingState::FullyShredded {
+ metadata: metadata.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => ShreddingState::PartiallyShredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ }
+ }
+}
+
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
Review Comment:
other Arrow Array apis don't return `Option`, they instead return the value directly requiring you to check `is_null` instead -- see https://docs.rs/arrow/latest/arrow/array/struct.PrimitiveArray.html#method.value for example
```rust
/// Consistently with other Arrow arrays types, this API requires you to
/// check for nulls first using [`Self::is_valid`].
```
I think the reason it to allow a better chance at LLVM vectorizing the code, which I don't think is likely relevant here.
We could make the Variant API deviate from this pattern (it is already different in several other ways) and return `Option<Variant>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257079494
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ ))),
+ }
+ }
+
+ /// Return a reference to the metadata field
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ match self {
+ ShreddingState::Unshredded { metadata, .. } => metadata,
+ ShreddingState::FullyShredded { metadata, .. } => metadata,
+ ShreddingState::PartiallyShredded { metadata, .. } => metadata,
+ }
+ }
+
+ /// Return a reference to the value field, if present
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ match self {
+ ShreddingState::Unshredded { value, .. } => Some(value),
+ ShreddingState::FullyShredded { .. } => None,
+ ShreddingState::PartiallyShredded { value, .. } => Some(value),
+ }
+ }
+
+ /// Return a reference to the typed_value field, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ match self {
+ ShreddingState::Unshredded { .. } => None,
+ ShreddingState::FullyShredded { typed_value, .. } => Some(typed_value),
+ ShreddingState::PartiallyShredded { typed_value, .. } => Some(typed_value),
+ }
+ }
+
+ /// Slice all the underlying arrays
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ match self {
+ ShreddingState::Unshredded { metadata, value } => ShreddingState::Unshredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ },
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => ShreddingState::FullyShredded {
+ metadata: metadata.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => ShreddingState::PartiallyShredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ }
+ }
+}
+
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
Review Comment:
> other Arrow Array apis don't return `Option`, they instead return the value directly requiring you to check `is_null` instead
For public API -- 100% agree. This is a private internal API tho, so it seems like we have room to do what we think makes the code simple/maintainable. Pulling important work inside the method instead of requiring all callers to remember it seems like a good example of that.
We can always change it if we discover it hurts performance or readability.
> I think the reason it to allow a better chance at LLVM vectorizing the code
AFAIK, arrow normally requires all array entries to be physically valid, even when null, so that one can perform columnar operations blindly and then just use the null masks to ignore the unwanted ones after the fact. Here, we're accessing is_null and value both inside the loop -- and the latter conditionally -- so I'd be very surprised if LLVM was willing to inject vectorization code that requires touching values the code said not to touch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2256926506
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
I thought more about this last night -- the reason I think we'll need special code other than `cast_to_variant` is that `cast_to_variant` does the downcast to typed array once, and then loops with the downcast array, where this code needs to downcast once and then convert 🤔 maybe someone can figure out something clever with macros
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3165502054
> Whops, I forgot to add some links. I will make a new PR for that
- https://github.com/apache/arrow-rs/pull/8092
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257079494
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ ))),
+ }
+ }
+
+ /// Return a reference to the metadata field
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ match self {
+ ShreddingState::Unshredded { metadata, .. } => metadata,
+ ShreddingState::FullyShredded { metadata, .. } => metadata,
+ ShreddingState::PartiallyShredded { metadata, .. } => metadata,
+ }
+ }
+
+ /// Return a reference to the value field, if present
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ match self {
+ ShreddingState::Unshredded { value, .. } => Some(value),
+ ShreddingState::FullyShredded { .. } => None,
+ ShreddingState::PartiallyShredded { value, .. } => Some(value),
+ }
+ }
+
+ /// Return a reference to the typed_value field, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ match self {
+ ShreddingState::Unshredded { .. } => None,
+ ShreddingState::FullyShredded { typed_value, .. } => Some(typed_value),
+ ShreddingState::PartiallyShredded { typed_value, .. } => Some(typed_value),
+ }
+ }
+
+ /// Slice all the underlying arrays
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ match self {
+ ShreddingState::Unshredded { metadata, value } => ShreddingState::Unshredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ },
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => ShreddingState::FullyShredded {
+ metadata: metadata.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => ShreddingState::PartiallyShredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ }
+ }
+}
+
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
Review Comment:
> other Arrow Array apis don't return `Option`, they instead return the value directly requiring you to check `is_null` instead
For public API -- 100% agree. This is a private internal API tho, so it seems like we have room to do what we think makes the code simple/maintainable. Pulling important work inside the method instead of requiring all callers to remember it seems like a good example of that.
We can always change it if we discover it hurts performance or readability.
> I think the reason it to allow a better chance at LLVM vectorizing the code
AFAIK, arrow normally requires all array entries to be physically valid, even when logically null. That way, one can perform columnar operations blindly and then just use the null masks to ignore the unwanted values after the fact. Here, we're accessing `is_null` and `value` both inside the loop -- and the latter conditionally -- so I'd be very surprised if LLVM was willing to inject vectorization code that requires touching values the code said not to touch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257329656
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +142,195 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded { typed_value, .. } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
Review Comment:
For a future `FullyShreddedAllNull` variant (neither `value` nor `typed_value` present), would we still need to store the `metadata` even tho it's never actually used? 🤔
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -93,35 +88,47 @@ impl VariantArray {
"Invalid VariantArray: requires StructArray as input".to_string(),
));
};
- // Ensure the StructArray has a metadata field of BinaryView
- let Some(metadata_field) = VariantArray::find_metadata_field(inner) else {
+ // Note the specification allows for any order so we must search by name
+
+ // Ensure the StructArray has a metadata field of BinaryView
+ let Some(metadata_field) = inner.column_by_name("metadata") else {
return Err(ArrowError::InvalidArgumentError(
"Invalid VariantArray: StructArray must contain a 'metadata' field".to_string(),
));
};
- if metadata_field.data_type() != &DataType::BinaryView {
+ let Some(metadata) = metadata_field.as_binary_view_opt() else {
return Err(ArrowError::NotYetImplemented(format!(
"VariantArray 'metadata' field must be BinaryView, got {}",
metadata_field.data_type()
)));
- }
- let Some(value_field) = VariantArray::find_value_field(inner) else {
- return Err(ArrowError::InvalidArgumentError(
- "Invalid VariantArray: StructArray must contain a 'value' field".to_string(),
- ));
};
- if value_field.data_type() != &DataType::BinaryView {
- return Err(ArrowError::NotYetImplemented(format!(
- "VariantArray 'value' field must be BinaryView, got {}",
- value_field.data_type()
- )));
- }
+
+ // Find the value field, if present
+ let value = inner
+ .column_by_name("value")
+ .map(|v| {
+ let Some(binary_view) = v.as_binary_view_opt() else {
+ return Err(ArrowError::NotYetImplemented(format!(
+ "VariantArray 'value' field must be BinaryView, got {}",
+ v.data_type()
+ )));
+ };
+ Ok(binary_view)
Review Comment:
Isn't this just
```suggestion
v.as_binary_view_opt().ok_or_else(|| {
ArrowError::NotYetImplemented(format!(
"VariantArray 'value' field must be BinaryView, got {}",
v.data_type()
));
})
```
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
+ let variant = variant_array.value(i);
+ let Some(value) = T::from_variant(&variant) else {
+ if self.cast_options.safe {
+ // safe mode: append null if we can't convert
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ } else {
+ return Err(ArrowError::CastError(format!(
+ "Failed to extract primitive of type {} from variant {:?} at path {:?}",
+ self.as_type.data_type(),
+ variant,
+ self.path
+ )));
+ }
+ };
+
+ nulls.append_non_null();
+ values.push(value)
+ } else {
+ // otherwise we have a typed value, so we can use it directly
+ nulls.append_non_null();
+ values.push(typed_value.value(i));
+ }
+ }
+
+ let nulls = nulls.finish();
+ let array = PrimitiveArray::<T>::new(values.into(), nulls)
+ .with_data_type(self.as_type.data_type().clone());
+ Ok(Arc::new(array))
+ }
+
+ fn fully_shredded(
+ &self,
+ _variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // if the types match exactly, we can just return the typed_value
+ if typed_value.data_type() == self.as_type.data_type() {
+ Ok(typed_value.clone())
+ } else {
+ // TODO: try to cast the typed_value to the desired type?
+ Err(ArrowError::NotYetImplemented(format!(
Review Comment:
I guess we could pursue three potential levels of casting:
1. Widening casts (based on type). E.g. if the type is int32, we can also handle int8, int16 and even decimal4 (if scale=0). The conversion is guaranteed to succeed losslessly.
2. Lossless narrowing casts (based on value). E.g. if the type is int32, we can still handle a small int64 value like 42. TBD what should happen if any one value failed to cast?
3. Converting casts. These are the lossy ones, e.g. converting 3.14f64 to 3i32, or 3i32 to string. I'm guessing we don't want to mess with these?
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
Review Comment:
It took a minute to realize that "partially shredded" here does _not_ refer to the [variant shredding spec](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding) definition:
> An object is partially shredded when the `value` is an object and the `typed_value` is a shredded object. Writers must not produce data where both `value` and `typed_value` are non-null, unless the Variant value is an object.
and
> The value column of a partially shredded object must never contain fields represented by the Parquet columns in `typed_value` (shredded fields). Readers may always assume that data is written correctly and that shredded fields in `typed_value` are not present in `value`.
This function seems to actually refer to "imperfectly shredded" columns, where we need a `value` column as a fallback for input values that fail to shred? As opposed to "perfectly shredded" columns that only need a `typed_value` column (or where the `value` column is all-null).
##########
parquet-variant-compute/src/variant_get/mod.rs:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow::{
+ array::{Array, ArrayRef},
+ compute::CastOptions,
+ error::Result,
+};
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::VariantPath;
+
+use crate::variant_array::ShreddingState;
+use crate::variant_get::output::instantiate_output_builder;
+use crate::VariantArray;
+
+mod output;
+
+/// Returns an array with the specified path extracted from the variant values.
+///
+/// The return array type depends on the `as_type` field of the options parameter
+/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
+/// to the specified path.
+/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
+ let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
+ ArrowError::InvalidArgumentError(
+ "expected a VariantArray as the input for variant_get".to_owned(),
+ )
+ })?;
+
+ // Create the output writer based on the specified output options
+ let output_builder = instantiate_output_builder(options.clone())?;
+
+ // Dispatch based on the shredding state of the input variant array
+ match variant_array.shredding_state() {
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => output_builder.partially_shredded(variant_array, metadata, value, typed_value),
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => output_builder.fully_shredded(variant_array, metadata, typed_value),
+ ShreddingState::Unshredded { metadata, value } => {
+ output_builder.unshredded(variant_array, metadata, value)
+ }
+ }
+}
+
+/// Controls the action of the variant_get kernel.
+#[derive(Debug, Clone, Default)]
+pub struct GetOptions<'a> {
+ /// What path to extract
+ pub path: VariantPath<'a>,
+ /// if `as_type` is None, the returned array will itself be a VariantArray.
+ ///
+ /// if `as_type` is `Some(type)` the field is returned as the specified type.
+ pub as_type: Option<FieldRef>,
Review Comment:
Presumably `None` and `Some(VariantType)` are equivalent?
If so, should we just default-initialize `as_type: FieldRef` to `VariantType`?
##########
parquet-variant-compute/src/variant_get/output/mod.rs:
##########
Review Comment:
Currently, the output builder seems to be fully column-oriented -- it assumes that all values for each leaf column are extracted in a tight loop. This can work for primitive builders, but nested builders will quickly run into pathing and efficiency problems.
I think we'll need to do something similar to the JSON builder, with a row-oriented approach where each level of a nested builder receives an already-constructed `Variant` for the current row and does a field extract for each child builder; child builders can then cast the result directly or recurse further as needed (based on their own type). And then the top-level builder call would construct a `Variant` for each row to kick-start the process.
But see the other comment -- to the extend that the shredding aligns nicely, we can hoist a subset of this per-row pathing of the append method up to columnar pathing of the builder's constructor and finalizer.
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
Ah, I think I see -- for a strongly-typed builder, the caller could do all the pathing and just pass in the correct leaf column here. But this is variant, so we'd actually need to extract the variant leaf value (using a path) on a per-row basis. We'd either have to do _all_ the pathing here (recursively), or caller would have to to extract the struct/array once and recurse on its fields. And the latter would require a row-oriented builder where this builder seems to be column-oriented.
##########
parquet-variant-compute/src/variant_get/mod.rs:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow::{
+ array::{Array, ArrayRef},
+ compute::CastOptions,
+ error::Result,
+};
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::VariantPath;
+
+use crate::variant_array::ShreddingState;
+use crate::variant_get::output::instantiate_output_builder;
+use crate::VariantArray;
+
+mod output;
+
+/// Returns an array with the specified path extracted from the variant values.
+///
+/// The return array type depends on the `as_type` field of the options parameter
+/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
+/// to the specified path.
+/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
+ let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
+ ArrowError::InvalidArgumentError(
+ "expected a VariantArray as the input for variant_get".to_owned(),
+ )
+ })?;
+
+ // Create the output writer based on the specified output options
+ let output_builder = instantiate_output_builder(options.clone())?;
+
+ // Dispatch based on the shredding state of the input variant array
+ match variant_array.shredding_state() {
Review Comment:
One (big) complexity here -- the requested type could differ wildly from how the input data are currently shredded.
Pathing-wise, we'd need to drill as deeply as possible into the variant array's physical shredding schema, following the requested path. In the happy path, the requested path was shredded and we just have to deal with the corresponding `(value, typed_value)` pair of columns.
<details>
<summary>Example</summary>
Suppose we're trying to extract `v:a.b::INT` from a `VariantArray` with the following physical layout:
```
v: VARIANT {
metadata: BINARY,
value: BINARY,
typed_value: STRUCT {
a: SHREDDED_VARIANT {
value: BINARY,
typed_value: STRUCT {
b: SHREDDED_VARIANT {
value: BINARY,
typed_value: INT,
},
},
},
},
}
```
Then we're actually doing `v.typed_value.a.typed_value.b::INT` -- the builder only has to worry about reconciling `v.typed_value.a.typed_value.b.value` with `v.typed_value.a.typed_value.b.typed_value` (which can be passed in directly at construction time)
This case has an even easier example, if the caller requested `v:a` (an untyped variant extract) -- in that case we simply combine `v.typed_value.a` with `v.metadata` into a new VariantArray -- no casting needed because `VariantArray` can capture the full structure `v:a` no matter how it was physically shredded.
</details>
In a less-happy path, we will hit a partially shredded object where the requested path element is _not_ available in `typed_value`, and we must go row-oriented from there (with the remaining path extraction performed on a per-row basis):
<details>
<summary>Example</summary>
Suppose we're trying to extract `v:a.b::INT` from a `VariantArray` with the following physical layout:
```
v: VARIANT {
metadata: BINARY,
value: BINARY,
typed_value: STRUCT {
a: SHREDDED_VARIANT {
value: BINARY,
typed_value: STRUCT {
x: SHREDDED_VARIANT {
value: BINARY,
typed_value: INT,
},
},
},
},
}
```
Then we're actually doing `v.typed_value.a:b::INT` -- we have to pass `v.typed_value.a.value` into the builder, which has to extract field `b` from every row.
</details>
In an unhappy path, the partial shredding of the input disagrees with the output's requested shredding, and we have to build up a struct from both shredded and unshredded paths.
<details>
<summary>Example</summary>
Suppose the requested schema asks for both `v:a.b::INT` _and_ `v:a.x::INT`, when the VariantArray's layout is:
```
v: VARIANT {
metadata: BINARY,
value: BINARY,
typed_value: STRUCT {
a: SHREDDED_VARIANT {
value: BINARY,
typed_value: STRUCT {
b: SHREDDED_VARIANT {
value: BINARY,
typed_value: INT,
},
},
},
},
}
```
Now we have to actually build the struct `v.a` with two fields by unioning shredded (`v.typed_value.a.typed_value.b::INT` and unshredded bits (`v.typed_value.a.value:x::INT`).
</details>
In a _really_ unhappy case, the caller requested a shredded variant type that disagrees with the input data's shredding. So some fields have to be shredded and others unshredded.
<details>
<summary>Example</summary>
Suppose we're trying to extract both `v:a` as a partially shredded variant with `x` shredded and `b` _not_ shredded... from a `VariantArray` where `b` is shredded and `x` is _not_ shredded:
```
v: VARIANT {
metadata: BINARY,
value: BINARY,
typed_value: STRUCT {
a: SHREDDED_VARIANT {
value: BINARY,
typed_value: STRUCT {
b: SHREDDED_VARIANT {
value: BINARY,
typed_value: INT,
},
},
},
},
}
```
Then not only would we need to path into `v.typed_value.a.typed_value.b::VARIANT` (which requires unshredding it and which produces an `o.typed_value.a.value_b` column), we'd also have to path into `v.typed_value.a.value::INT` (which requires shredding it and which contributes to _both_ `o.typed_value.a.value_x` _and_ `o.typed_value.a.typed_value` columns). And then we have to variant-union `o.typed_value.a.value_b` and `o.typed_value.a.value_x`columns into a single `o.typed_value.a.value` column, and pack everything into a new `VariantArray`.
</details>
##########
parquet-variant-compute/src/variant_get/mod.rs:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow::{
+ array::{Array, ArrayRef},
+ compute::CastOptions,
+ error::Result,
+};
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::VariantPath;
+
+use crate::variant_array::ShreddingState;
+use crate::variant_get::output::instantiate_output_builder;
+use crate::VariantArray;
+
+mod output;
+
+/// Returns an array with the specified path extracted from the variant values.
+///
+/// The return array type depends on the `as_type` field of the options parameter
+/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
+/// to the specified path.
+/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
+ let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
+ ArrowError::InvalidArgumentError(
+ "expected a VariantArray as the input for variant_get".to_owned(),
+ )
+ })?;
+
+ // Create the output writer based on the specified output options
+ let output_builder = instantiate_output_builder(options.clone())?;
+
+ // Dispatch based on the shredding state of the input variant array
+ match variant_array.shredding_state() {
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => output_builder.partially_shredded(variant_array, metadata, value, typed_value),
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => output_builder.fully_shredded(variant_array, metadata, typed_value),
+ ShreddingState::Unshredded { metadata, value } => {
+ output_builder.unshredded(variant_array, metadata, value)
+ }
+ }
+}
+
+/// Controls the action of the variant_get kernel.
+#[derive(Debug, Clone, Default)]
+pub struct GetOptions<'a> {
+ /// What path to extract
+ pub path: VariantPath<'a>,
+ /// if `as_type` is None, the returned array will itself be a VariantArray.
+ ///
+ /// if `as_type` is `Some(type)` the field is returned as the specified type.
+ pub as_type: Option<FieldRef>,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ pub cast_options: CastOptions<'a>,
+}
+
+impl<'a> GetOptions<'a> {
+ /// Construct default options to get the specified path as a variant.
+ pub fn new() -> Self {
+ Default::default()
+ }
+
+ /// Construct options to get the specified path as a variant.
+ pub fn new_with_path(path: VariantPath<'a>) -> Self {
Review Comment:
I guess path is optional (defaults to empty) because a leaf extract of a perfectly shredded variant doesn't need any internal pathing -- whoever constructs the builder just passes the leaf columns and the builder only needs to cast them appropriately.
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -186,13 +340,11 @@ impl Array for VariantArray {
}
fn slice(&self, offset: usize, length: usize) -> ArrayRef {
- let slice = self.inner.slice(offset, length);
- let met = self.metadata_ref.slice(offset, length);
- let val = self.value_ref.slice(offset, length);
+ let inner = self.inner.slice(offset, length);
+ let shredding_state = self.shredding_state.slice(offset, length);
Arc::new(Self {
Review Comment:
Out of curiosity, would we actually expect a redundant `slice` call to be notably more expensive than `clone`?
Seems like they'd do the same thing?
<details>
For example, `BooleanArray::slice` is:
```rust
pub fn slice(&self, offset: usize, length: usize) -> Self {
Self {
values: self.values.slice(offset, length),
nulls: self.nulls.as_ref().map(|n| n.slice(offset, length)),
}
}
```
with `BooleanBuffer::slice`:
```rust
pub fn slice(&self, offset: usize, len: usize) -> Self {
assert!(
offset.saturating_add(len) <= self.len,
"the length + offset of the sliced BooleanBuffer cannot exceed the existing length"
);
Self {
buffer: self.buffer.clone(),
offset: self.offset + offset,
len,
}
}
```
Assuming the compiler inlines things as aggressively as it usually does, it seems like the net difference would be the bounds check and other offset arithmetic.
</details>
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -93,35 +88,45 @@ impl VariantArray {
"Invalid VariantArray: requires StructArray as input".to_string(),
));
};
- // Ensure the StructArray has a metadata field of BinaryView
- let Some(metadata_field) = VariantArray::find_metadata_field(inner) else {
+ // Note the specification allows for any order so we must search by name
+
+ // Ensure the StructArray has a metadata field of BinaryView
+ let Some(metadata_field) = inner.column_by_name("metadata") else {
return Err(ArrowError::InvalidArgumentError(
"Invalid VariantArray: StructArray must contain a 'metadata' field".to_string(),
));
};
- if metadata_field.data_type() != &DataType::BinaryView {
+ let Some(metadata) = metadata_field.as_binary_view_opt() else {
return Err(ArrowError::NotYetImplemented(format!(
"VariantArray 'metadata' field must be BinaryView, got {}",
metadata_field.data_type()
)));
- }
- let Some(value_field) = VariantArray::find_value_field(inner) else {
- return Err(ArrowError::InvalidArgumentError(
- "Invalid VariantArray: StructArray must contain a 'value' field".to_string(),
- ));
};
- if value_field.data_type() != &DataType::BinaryView {
- return Err(ArrowError::NotYetImplemented(format!(
- "VariantArray 'value' field must be BinaryView, got {}",
- value_field.data_type()
- )));
- }
+
+ // Find the value field, if present
+ let value_field = inner.column_by_name("value");
+ let value = value_field
+ .map(|v| match v.as_binary_view_opt() {
+ Some(bv) => Ok(bv),
+ None => Err(ArrowError::NotYetImplemented(format!(
+ "VariantArray 'value' field must be BinaryView, got {}",
+ v.data_type()
+ ))),
+ })
+ .transpose()?;
Review Comment:
Oops, I meant `ok_or_else`...
##########
parquet-variant-compute/src/variant_get/mod.rs:
##########
@@ -0,0 +1,430 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow::{
+ array::{Array, ArrayRef},
+ compute::CastOptions,
+ error::Result,
+};
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::VariantPath;
+
+use crate::variant_array::ShreddingState;
+use crate::variant_get::output::instantiate_output_builder;
+use crate::VariantArray;
+
+mod output;
+
+/// Returns an array with the specified path extracted from the variant values.
+///
+/// The return array type depends on the `as_type` field of the options parameter
+/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
+/// to the specified path.
+/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
+ let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
+ ArrowError::InvalidArgumentError(
+ "expected a VariantArray as the input for variant_get".to_owned(),
+ )
+ })?;
+
+ // Create the output writer based on the specified output options
+ let output_builder = instantiate_output_builder(options.clone())?;
+
+ // Dispatch based on the shredding state of the input variant array
+ match variant_array.shredding_state() {
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => output_builder.partially_shredded(variant_array, metadata, value, typed_value),
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => output_builder.fully_shredded(variant_array, metadata, typed_value),
+ ShreddingState::Unshredded { metadata, value } => {
+ output_builder.unshredded(variant_array, metadata, value)
+ }
+ }
+}
+
+/// Controls the action of the variant_get kernel.
+#[derive(Debug, Clone, Default)]
+pub struct GetOptions<'a> {
+ /// What path to extract
+ pub path: VariantPath<'a>,
+ /// if `as_type` is None, the returned array will itself be a VariantArray.
+ ///
+ /// if `as_type` is `Some(type)` the field is returned as the specified type.
+ pub as_type: Option<FieldRef>,
Review Comment:
Oh... but variant is not a single type... every possible shredding (or lack thereof) is a different physical `DataType`. So by passing `None`, we request to keep whatever structure the underlying path already has; by passing `Some(... variant type...)` we're requesting to [un|re]shred as necessary to make it fit the new shredding schema.
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
Why would there be a path at all for this primitive output builder, sorry?
Seems like that only matters for array and object builders?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261168165
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
For example, suppose that the user requested path `v:a.b.c.d::INT`, and that `a` and `b` were shredded but not `c`. Then the corresponding builder would take `v.typed_value.a.typed_value.b.value` as its input column, with `c.d::INT` as the path to extract.
Meanwhile, if the user also requested `v:a.b.x.y::INT` which happens to be perfectly shredded, then the corresponding "builder" would take `v.typed_value.a.typed_value.b.typed_value.x.typed_value.y.typed_value` as its input column. It would not need to extract anything (so its per-row operation is a no-op) and its finalizer just needs to return the unchanged input `ArrayRef` so the parent builder for `v:a.b` can "snap" it into the correct place while assembling the `StructArray` that represents `v:a.b`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3153144846
> @scovich @carpecodeum and @Samyak2 this PR I think is ready for review
Whoop, I somehow missed the notify. I'll try to take a look in the next day or two!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "Samyak2 (via GitHub)" <gi...@apache.org>.
Samyak2 commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2255052328
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
That's unfortunate. Maybe a separate issue for that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "carpecodeum (via GitHub)" <gi...@apache.org>.
carpecodeum commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3160659989
looks great!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP [Variant] Strawman `variant_get` and Shredded variants [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2249343322
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -44,27 +45,90 @@ use std::sync::Arc;
/// [document]: https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?usp=sharing
#[derive(Debug)]
pub struct VariantArray {
- /// StructArray of up to three fields:
- ///
- /// 1. A required field named `metadata` which is binary, large_binary, or
- /// binary_view
- ///
- /// 2. An optional field named `value` that is binary, large_binary, or
- /// binary_view
- ///
- /// 3. An optional field named `typed_value` which can be any primitive type
- /// or be a list, large_list, list_view or struct
- ///
- /// NOTE: It is also permissible for the metadata field to be
- /// Dictionary-Encoded, preferably (but not required) with an index type of
- /// int8.
+ /// Reference to the underlying StructArray
inner: StructArray,
- /// Reference to the metadata column of inner
- metadata_ref: ArrayRef,
+ /// how is this variant array shredded?
+ shredding_state: ShreddingState,
+}
+
+/// Variant arrays can be shredded in one of three states, encoded here
Review Comment:
With this enum it is easier to know what the state of any particular array is
##########
parquet-variant-compute/src/variant_get/output/mod.rs:
##########
@@ -0,0 +1,87 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+mod primitive;
+mod variant;
+
+use crate::variant_get::output::primitive::PrimitiveOutputBuilder;
+use crate::variant_get::output::variant::VariantOutputBuilder;
+use crate::variant_get::GetOptions;
+use crate::VariantArray;
+use arrow::array::{ArrayRef, BinaryViewArray};
+use arrow::datatypes::Int32Type;
+use arrow::error::Result;
+use arrow_schema::{ArrowError, DataType};
+
+/// This trait represents something that gets the output of the variant_get kernel.
+///
+/// For example, there are specializations for writing the output as a VariantArray,
+/// or as a specific type (e.g. Int32Array).
+///
+/// See [`instantiate_output_builder`] to create an instance of this trait.
+pub(crate) trait OutputBuilder {
+ /// create output for a shredded variant array
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ metadata: &BinaryViewArray,
+ value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> Result<ArrayRef>;
+
+ /// output for a perfectly shredded variant array
+ fn fully_shredded(
+ &self,
+ variant_array: &VariantArray,
+ metadata: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> Result<ArrayRef>;
+
+ /// write out an unshredded variant array
+ fn unshredded(
+ &self,
+ variant_array: &VariantArray,
+ metadata: &BinaryViewArray,
+ value_field: &BinaryViewArray,
+ ) -> Result<ArrayRef>;
+}
+
+pub(crate) fn instantiate_output_builder<'a>(
Review Comment:
the idea here is that this creates an instance of `OutputBuilder` based on the requested output type
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -1,180 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
I actually left most of this file the same but renamed it to `variant_get/mod.rs`, despite how github is rendering it
##########
parquet-variant-compute/src/variant_get/mod.rs:
##########
@@ -0,0 +1,431 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow::{
+ array::{Array, ArrayRef},
+ compute::CastOptions,
+ error::Result,
+};
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::VariantPath;
+
+use crate::variant_array::ShreddingState;
+use crate::variant_get::output::instantiate_output_builder;
+use crate::VariantArray;
+
+mod output;
+
+/// Returns an array with the specified path extracted from the variant values.
+///
+/// The return array type depends on the `as_type` field of the options parameter
+/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
+/// to the specified path.
+/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
+ let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
+ ArrowError::InvalidArgumentError(
+ "expected a VariantArray as the input for variant_get".to_owned(),
+ )
+ })?;
+
+ // Create the output writer based on the specified output options
Review Comment:
The core design for get is here: Different potential builders depending on the type of the output array -- which gives us a place to put special code for each output array type, this is the `OutputBuilder` trait -- perhaps someone can come up with a better name.
##########
parquet-variant-compute/src/variant_get/mod.rs:
##########
@@ -0,0 +1,431 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow::{
+ array::{Array, ArrayRef},
+ compute::CastOptions,
+ error::Result,
+};
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::VariantPath;
+
+use crate::variant_array::ShreddingState;
+use crate::variant_get::output::instantiate_output_builder;
+use crate::VariantArray;
+
+mod output;
+
+/// Returns an array with the specified path extracted from the variant values.
+///
+/// The return array type depends on the `as_type` field of the options parameter
+/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
+/// to the specified path.
+/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
+ let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
+ ArrowError::InvalidArgumentError(
+ "expected a VariantArray as the input for variant_get".to_owned(),
+ )
+ })?;
+
+ // Create the output writer based on the specified output options
+ let output_builder = instantiate_output_builder(options.clone())?;
+
+ // Dispatch based on the shredding state of the input variant array
+ // TODO make this an enum on VariantArray (e.g ShreddingState)
+ match variant_array.shredding_state() {
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => output_builder.partially_shredded(variant_array, metadata, value, typed_value),
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => output_builder.fully_shredded(variant_array, metadata, typed_value),
+ ShreddingState::Unshredded { metadata, value } => {
+ output_builder.unshredded(variant_array, metadata, value)
+ }
+ }
+}
+
+/// Controls the action of the variant_get kernel.
+#[derive(Debug, Clone, Default)]
+pub struct GetOptions<'a> {
+ /// What path to extract
+ pub path: VariantPath<'a>,
+ /// if `as_type` is None, the returned array will itself be a VariantArray.
+ ///
+ /// if `as_type` is `Some(type)` the field is returned as the specified type.
+ pub as_type: Option<FieldRef>,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ pub cast_options: CastOptions<'a>,
+}
+
+impl<'a> GetOptions<'a> {
+ /// Construct default options to get the specified path as a variant.
+ pub fn new() -> Self {
+ Default::default()
+ }
+
+ /// Construct options to get the specified path as a variant.
+ pub fn new_with_path(path: VariantPath<'a>) -> Self {
+ Self {
+ path,
+ as_type: None,
+ cast_options: Default::default(),
+ }
+ }
+
+ /// Specify the type to return.
+ pub fn with_as_type(mut self, as_type: Option<FieldRef>) -> Self {
+ self.as_type = as_type;
+ self
+ }
+
+ /// Specify the cast options to use when casting to the specified type.
+ pub fn with_cast_options(mut self, cast_options: CastOptions<'a>) -> Self {
+ self.cast_options = cast_options;
+ self
+ }
+}
+
+#[cfg(test)]
+mod test {
+ use std::sync::Arc;
+
+ use arrow::array::{Array, ArrayRef, BinaryViewArray, Int32Array, StringArray, StructArray};
+ use arrow::buffer::NullBuffer;
+ use arrow::compute::CastOptions;
+ use arrow_schema::{DataType, Field, FieldRef, Fields};
+ use parquet_variant::{Variant, VariantPath};
+
+ use crate::batch_json_string_to_variant;
+ use crate::VariantArray;
+
+ use super::{variant_get, GetOptions};
+
+ fn single_variant_get_test(input_json: &str, path: VariantPath, expected_json: &str) {
+ // Create input array from JSON string
+ let input_array_ref: ArrayRef = Arc::new(StringArray::from(vec![Some(input_json)]));
+ let input_variant_array_ref: ArrayRef =
+ Arc::new(batch_json_string_to_variant(&input_array_ref).unwrap());
+
+ let result =
+ variant_get(&input_variant_array_ref, GetOptions::new_with_path(path)).unwrap();
+
+ // Create expected array from JSON string
+ let expected_array_ref: ArrayRef = Arc::new(StringArray::from(vec![Some(expected_json)]));
+ let expected_variant_array = batch_json_string_to_variant(&expected_array_ref).unwrap();
+
+ let result_array: &VariantArray = result.as_any().downcast_ref().unwrap();
+ assert_eq!(
+ result_array.len(),
+ 1,
+ "Expected result array to have length 1"
+ );
+ assert!(
+ result_array.nulls().is_none(),
+ "Expected no nulls in result array"
+ );
+ let result_variant = result_array.value(0);
+ let expected_variant = expected_variant_array.value(0);
+ assert_eq!(
+ result_variant, expected_variant,
+ "Result variant does not match expected variant"
+ );
+ }
+
+ #[test]
+ fn get_primitive_variant_field() {
+ single_variant_get_test(
+ r#"{"some_field": 1234}"#,
+ VariantPath::from("some_field"),
+ "1234",
+ );
+ }
+
+ #[test]
+ fn get_primitive_variant_list_index() {
+ single_variant_get_test("[1234, 5678]", VariantPath::from(0), "1234");
+ }
+
+ #[test]
+ fn get_primitive_variant_inside_object_of_object() {
+ single_variant_get_test(
+ r#"{"top_level_field": {"inner_field": 1234}}"#,
+ VariantPath::from("top_level_field").join("inner_field"),
+ "1234",
+ );
+ }
+
+ #[test]
+ fn get_primitive_variant_inside_list_of_object() {
+ single_variant_get_test(
+ r#"[{"some_field": 1234}]"#,
+ VariantPath::from(0).join("some_field"),
+ "1234",
+ );
+ }
+
+ #[test]
+ fn get_primitive_variant_inside_object_of_list() {
+ single_variant_get_test(
+ r#"{"some_field": [1234]}"#,
+ VariantPath::from("some_field").join(0),
+ "1234",
+ );
+ }
+
+ #[test]
+ fn get_complex_variant() {
+ single_variant_get_test(
+ r#"{"top_level_field": {"inner_field": 1234}}"#,
+ VariantPath::from("top_level_field"),
+ r#"{"inner_field": 1234}"#,
+ );
+ }
+
+ /// Shredding: extract a value as a VariantArray
Review Comment:
here are the new shredded tests
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,76 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
- pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ pub fn value(&self, index: usize) -> Option<Variant> {
Review Comment:
This is the first part -- accessing a single Variant value from a potentially shredded VariantArray
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
+ let Some(variant) = variant_array.value(i) else {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ };
+
+ let Some(value) = T::from_variant(&variant) else {
+ if self.cast_options.safe {
+ // safe mode: append null if we can't convert
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ } else {
+ return Err(ArrowError::CastError(format!(
+ "Failed to extract primitive of type {} from variant {:?} at path {:?}",
+ self.as_type.data_type(),
+ variant,
+ self.path
+ )));
+ }
+ };
+
+ nulls.append_non_null();
+ values.push(value)
+ } else {
+ // otherwise we have a typed value, so we can use it directly
+ nulls.append_non_null();
+ values.push(typed_value.value(i));
+ }
+ }
+
+ let nulls = nulls.finish();
+ let array = PrimitiveArray::<T>::new(values.into(), nulls)
+ .with_data_type(self.as_type.data_type().clone());
+ Ok(Arc::new(array))
+ }
+
+ fn fully_shredded(
+ &self,
+ _variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // if the types match exactly, we can just return the typed_value
+ if typed_value.data_type() == self.as_type.data_type() {
Review Comment:
here is the fast path if we have perfectly shredded value
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
Review Comment:
This is the code that converts a shredded variant into a typed output -- It can probably be made quite a bit faster with some more deliberate vectorization, but I think it is relatively simple and functional
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
Review Comment:
this one is a bit complicated due to generics but I think it will work for all primitive types (ints, floats, etc)
##########
parquet-variant-compute/src/variant_get/output/variant.rs:
##########
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::{VariantArray, VariantArrayBuilder};
+use arrow::array::{Array, ArrayRef, AsArray, BinaryViewArray};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, DataType};
+use parquet_variant::{Variant, VariantPath};
+use std::sync::Arc;
+
+/// Outputs VariantArrays
+pub(super) struct VariantOutputBuilder<'a> {
+ /// What path to extract
+ path: VariantPath<'a>,
+}
+
+impl<'a> VariantOutputBuilder<'a> {
+ pub(super) fn new(path: VariantPath<'a>) -> Self {
+ Self { path }
+ }
+}
+
+impl<'a> OutputBuilder for VariantOutputBuilder<'a> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ // TODO(perf): can reuse the metadata field here to avoid re-creating it
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // in this case dispatch on the typed_value and
+ // TODO macro'ize this using downcast! to handle all other primitive types
+ // TODO(perf): avoid builders entirely (and write the raw variant directly as we know the metadata is the same)
+ let mut array_builder = VariantArrayBuilder::new(variant_array.len());
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let primitive_array = typed_value.as_primitive::<Int32Type>();
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ array_builder.append_null();
+ continue;
+ }
+
+ if typed_value.is_null(i) {
+ // fall back to the value (variant) field
+ // (TODO could copy the variant bytes directly)
+ if let Some(value) = variant_array.value(i) {
+ array_builder.append_variant(value);
+ } else {
+ // this shouldn't happen, but if it does, append null
+ array_builder.append_null();
+ }
+ continue;
+ }
+
+ // otherwise we have a typed value, so we can use it directly
+ let int_value = primitive_array.value(i);
+ array_builder.append_variant(Variant::from(int_value));
+ }
+ }
+ dt => {
+ return Err(ArrowError::NotYetImplemented(format!(
+ "variant_get fully_shredded with typed_value={dt} is not implemented yet",
+ )));
+ }
+ };
+ Ok(Arc::new(array_builder.build()))
+ }
+
+ fn fully_shredded(
+ &self,
+ variant_array: &VariantArray,
+ // TODO(perf): can reuse the metadata field here to avoid re-creating it
+ _metadata: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // in this case dispatch on the typed_value and
+ // TODO macro'ize this using downcast! to handle all other primitive types
+ // TODO(perf): avoid builders entirely (and write the raw variant directly as we know the metadata is the same)
+ let mut array_builder = VariantArrayBuilder::new(variant_array.len());
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let primitive_array = typed_value.as_primitive::<Int32Type>();
+ for i in 0..variant_array.len() {
+ if primitive_array.is_null(i) {
+ array_builder.append_null();
+ continue;
+ }
+
+ let int_value = primitive_array.value(i);
+ array_builder.append_variant(Variant::from(int_value));
+ }
+ }
+ dt => {
+ return Err(ArrowError::NotYetImplemented(format!(
+ "variant_get fully_shredded with typed_value={dt} is not implemented yet",
+ )));
+ }
+ };
+ Ok(Arc::new(array_builder.build()))
+ }
+
+ fn unshredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ ) -> arrow::error::Result<ArrayRef> {
+ let mut builder = VariantArrayBuilder::new(variant_array.len());
Review Comment:
This is the case that is handled on `main`
##########
parquet-variant-compute/src/variant_get/output/variant.rs:
##########
@@ -0,0 +1,153 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::{VariantArray, VariantArrayBuilder};
+use arrow::array::{Array, ArrayRef, AsArray, BinaryViewArray};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, DataType};
+use parquet_variant::{Variant, VariantPath};
+use std::sync::Arc;
+
+/// Outputs VariantArrays
+pub(super) struct VariantOutputBuilder<'a> {
+ /// What path to extract
+ path: VariantPath<'a>,
+}
+
+impl<'a> VariantOutputBuilder<'a> {
+ pub(super) fn new(path: VariantPath<'a>) -> Self {
+ Self { path }
+ }
+}
+
+impl<'a> OutputBuilder for VariantOutputBuilder<'a> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ // TODO(perf): can reuse the metadata field here to avoid re-creating it
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // in this case dispatch on the typed_value and
+ // TODO macro'ize this using downcast! to handle all other primitive types
+ // TODO(perf): avoid builders entirely (and write the raw variant directly as we know the metadata is the same)
+ let mut array_builder = VariantArrayBuilder::new(variant_array.len());
Review Comment:
This is the logic to reconstruct `Variant`s from typed values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "carpecodeum (via GitHub)" <gi...@apache.org>.
carpecodeum commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2249639145
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -186,13 +340,11 @@ impl Array for VariantArray {
}
fn slice(&self, offset: usize, length: usize) -> ArrayRef {
- let slice = self.inner.slice(offset, length);
- let met = self.metadata_ref.slice(offset, length);
- let val = self.value_ref.slice(offset, length);
+ let inner = self.inner.slice(offset, length);
+ let shredding_state = self.shredding_state.slice(offset, length);
Arc::new(Self {
Review Comment:
can we avoid unnecessary allocations for common cases, like -
```
let shredding_state = match (&self.shredding_state, offset, length) {
// Fast path: no slice needed for full array
(state, 0, len) if len == self.len() => state.clone(),
// Fast path: uniform shredding doesn't need slicing
(ShreddingState::None, _, _) => ShreddingState::None,
(ShreddingState::All, _, _) => ShreddingState::All,
// Only slice for mixed case
(ShreddingState::Mixed(bitmap), offset, length) => {
ShreddingState::Mixed(bitmap.slice(offset, length))
}
};
```
##########
parquet-variant-compute/src/from_json.rs:
##########
@@ -69,8 +69,8 @@ mod test {
let array_ref: ArrayRef = Arc::new(input);
let variant_array = batch_json_string_to_variant(&array_ref).unwrap();
- let metadata_array = variant_array.metadata_field().as_binary_view();
- let value_array = variant_array.value_field().as_binary_view();
+ let metadata_array = variant_array.metadata_field();
+ let value_array = variant_array.value_field().expect("value field");
Review Comment:
Using expect() will cause panics. Since this is a library function, shouldn't it return Result<VariantArray, ArrowError> and let callers handle the error gracefully? Something like
```
let value_array = variant_array.value_field()
.ok_or_else(|| ArrowError::InvalidArgumentError(
"value field not available - array may be fully shredded".to_string()
))?;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "Samyak2 (via GitHub)" <gi...@apache.org>.
Samyak2 commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2249924930
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -79,12 +143,22 @@ impl VariantArray {
/// # Errors:
/// - If the `StructArray` does not contain the required fields
///
- /// # Current support
- /// This structure does not (yet) support the full Arrow Variant Array specification.
+ /// # Requirements of the `StructArray`
+ ///
+ /// 1. A required field named `metadata` which is binary, large_binary, or
+ /// binary_view
///
- /// Only `StructArrays` with `metadata` and `value` fields that are
- /// [`BinaryViewArray`] are supported. Shredded values are not currently supported
- /// nor are using types other than `BinaryViewArray`
+ /// 2. An optional field named `value` that is binary, large_binary, or
+ /// binary_view
+ ///
+ /// 3. An optional field named `typed_value` which can be any primitive type
+ /// or be a list, large_list, list_view or struct
+ ///
+ /// NOTE: It is also permissible for the metadata field to be
+ /// Dictionary-Encoded, preferably (but not required) with an index type of
+ /// int8.
+ ///
+ /// Currently, only [`BinaryViewArray`] are supported.
Review Comment:
Double space is intentional?
```suggestion
/// Currently, only [`BinaryViewArray`] are supported.
```
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
I have one concern with this. I think the path-ing should be done independent of the output builder. Paths are only valid on objects/lists inside the variant. I don't see why every type's output builder needs to be concerned with the path.
Even for shredded objects, we can bring the nested value/typed_value fields up and pass it along like a top-level shredded variant array.
What do you think? Let me know if I missed something (I wasn't actively following the discussions around this).
##########
parquet-variant-compute/src/variant_get/mod.rs:
##########
@@ -0,0 +1,431 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow::{
+ array::{Array, ArrayRef},
+ compute::CastOptions,
+ error::Result,
+};
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::VariantPath;
+
+use crate::variant_array::ShreddingState;
+use crate::variant_get::output::instantiate_output_builder;
+use crate::VariantArray;
+
+mod output;
+
+/// Returns an array with the specified path extracted from the variant values.
+///
+/// The return array type depends on the `as_type` field of the options parameter
+/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
+/// to the specified path.
+/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
+ let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
+ ArrowError::InvalidArgumentError(
+ "expected a VariantArray as the input for variant_get".to_owned(),
+ )
+ })?;
+
+ // Create the output writer based on the specified output options
+ let output_builder = instantiate_output_builder(options.clone())?;
+
+ // Dispatch based on the shredding state of the input variant array
+ // TODO make this an enum on VariantArray (e.g ShreddingState)
Review Comment:
This TODO seems to be done?
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
This could be useful as a (public) function on its own. For example, this could be used to cast columns to variant like in `some_column::variant`
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -93,35 +167,64 @@ impl VariantArray {
"Invalid VariantArray: requires StructArray as input".to_string(),
));
};
- // Ensure the StructArray has a metadata field of BinaryView
- let Some(metadata_field) = VariantArray::find_metadata_field(inner) else {
+ // Note the specification allows for any order so we must search by name
+
+ // Ensure the StructArray has a metadata field of BinaryView
+ let Some(metadata_field) = inner.column_by_name("metadata") else {
return Err(ArrowError::InvalidArgumentError(
"Invalid VariantArray: StructArray must contain a 'metadata' field".to_string(),
));
};
- if metadata_field.data_type() != &DataType::BinaryView {
+ let Some(metadata) = metadata_field.as_binary_view_opt() else {
return Err(ArrowError::NotYetImplemented(format!(
"VariantArray 'metadata' field must be BinaryView, got {}",
metadata_field.data_type()
)));
- }
- let Some(value_field) = VariantArray::find_value_field(inner) else {
- return Err(ArrowError::InvalidArgumentError(
- "Invalid VariantArray: StructArray must contain a 'value' field".to_string(),
- ));
};
- if value_field.data_type() != &DataType::BinaryView {
- return Err(ArrowError::NotYetImplemented(format!(
- "VariantArray 'value' field must be BinaryView, got {}",
- value_field.data_type()
- )));
- }
+
+ // Find the value field, if present
+ let value_field = inner.column_by_name("value");
+ let value = value_field
+ .map(|v| match v.as_binary_view_opt() {
+ Some(bv) => Ok(bv),
+ None => Err(ArrowError::NotYetImplemented(format!(
+ "VariantArray 'value' field must be BinaryView, got {}",
+ v.data_type()
+ ))),
+ })
+ .transpose()?;
+
+ // Find the typed_value field, if present
+ let typed_value = inner.column_by_name("typed_value");
+
+ // Note these clones are cheap, they just bump the ref count
+ let inner = inner.clone();
+ let metadata = metadata.clone();
+ let value = value.cloned();
+ let typed_value = typed_value.cloned();
+
+ let shredding_state = match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ },
+ (metadata, Some(value), None) => ShreddingState::Unshredded { metadata, value },
+ (metadata, None, Some(typed_value)) => ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ },
+ (_metadata_field, None, None) => {
+ return Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ )));
+ }
+ };
Review Comment:
Could this be a `try_new` constructor for `ShreddingState`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3155834568
To be clear, my ideal outcome would be:
1. We agree that the structure of this PR is ok (we can also change it in future PRs, but I want to make sure it looks ok before merging)
2. I will file a bunch of follow on tickets to break down the work of filling out the variant feature set
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261110415
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
Review Comment:
This is an excellent call -- I have updated the names to match the shredding specification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261250591
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
> That's unfortunate. Maybe a separate issue for that?
- Filed https://github.com/apache/arrow-rs/issues/8091
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3146689723
@scovich @carpecodeum and @Samyak2 this PR I think is ready for review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257063486
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
Review Comment:
> The reason I first introduced the enum was that in `variant_get` I needed to dispatch to different methods based on the type of shredding, and I found it myself really wanting to have names for them rather than just the combinations of value and typed value.
I guess... but it seems like there's so much code just managing the resulting enum variants I question whether it's a net win. Like the getters for metadata, value, etc. or the above code that shrinks from 25 lines to seven while handling more cases.
Clearly either way can be made correct, tho, so I guess it's a matter of preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2256889454
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
Review Comment:
My proposal is to leave the ShreddingState enum in and I will file a follow on ticket to support the `none-none` case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257480706
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
> Basically, a macro that just does the array casting and allows the caller to inject whatever they want. This site here would inject a scalar lookup, while the cast kernel would inject a for-loop.
I think this is a good idea. I am not sure we can use the `downcast_to_primitive` as it dispatches based on underlying primtive type (e.g. you get an `i128` for a decimal, but don't get precision and scale). Maybe someone else can figure out how to make it work
We could maybe make our own macro variant specific macro though 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3165384068
This implementation is far from complete, but I don't see any objections to the fundamental structure.
So, let's merge this one in and work on improving the situation in follow on PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2254333332
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -79,12 +143,22 @@ impl VariantArray {
/// # Errors:
/// - If the `StructArray` does not contain the required fields
///
- /// # Current support
- /// This structure does not (yet) support the full Arrow Variant Array specification.
+ /// # Requirements of the `StructArray`
+ ///
+ /// 1. A required field named `metadata` which is binary, large_binary, or
+ /// binary_view
///
- /// Only `StructArrays` with `metadata` and `value` fields that are
- /// [`BinaryViewArray`] are supported. Shredded values are not currently supported
- /// nor are using types other than `BinaryViewArray`
+ /// 2. An optional field named `value` that is binary, large_binary, or
+ /// binary_view
+ ///
+ /// 3. An optional field named `typed_value` which can be any primitive type
+ /// or be a list, large_list, list_view or struct
+ ///
+ /// NOTE: It is also permissible for the metadata field to be
+ /// Dictionary-Encoded, preferably (but not required) with an index type of
+ /// int8.
+ ///
+ /// Currently, only [`BinaryViewArray`] are supported.
Review Comment:
I don't think so -- thank you
##########
parquet-variant-compute/src/variant_get/mod.rs:
##########
@@ -0,0 +1,431 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+use arrow::{
+ array::{Array, ArrayRef},
+ compute::CastOptions,
+ error::Result,
+};
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::VariantPath;
+
+use crate::variant_array::ShreddingState;
+use crate::variant_get::output::instantiate_output_builder;
+use crate::VariantArray;
+
+mod output;
+
+/// Returns an array with the specified path extracted from the variant values.
+///
+/// The return array type depends on the `as_type` field of the options parameter
+/// 1. `as_type: None`: a VariantArray is returned. The values in this new VariantArray will point
+/// to the specified path.
+/// 2. `as_type: Some(<specific field>)`: an array of the specified type is returned.
+pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
+ let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
+ ArrowError::InvalidArgumentError(
+ "expected a VariantArray as the input for variant_get".to_owned(),
+ )
+ })?;
+
+ // Create the output writer based on the specified output options
+ let output_builder = instantiate_output_builder(options.clone())?;
+
+ // Dispatch based on the shredding state of the input variant array
+ // TODO make this an enum on VariantArray (e.g ShreddingState)
Review Comment:
good call. Removed
##########
parquet-variant-compute/src/from_json.rs:
##########
@@ -69,8 +69,8 @@ mod test {
let array_ref: ArrayRef = Arc::new(input);
let variant_array = batch_json_string_to_variant(&array_ref).unwrap();
- let metadata_array = variant_array.metadata_field().as_binary_view();
- let value_array = variant_array.value_field().as_binary_view();
+ let metadata_array = variant_array.metadata_field();
+ let value_array = variant_array.value_field().expect("value field");
Review Comment:
You are right that `expect` causes a panic. However I think this code is part of the test `test_batch_json_string_to_variant` so it is probably ok to panic if something changes
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -93,35 +167,64 @@ impl VariantArray {
"Invalid VariantArray: requires StructArray as input".to_string(),
));
};
- // Ensure the StructArray has a metadata field of BinaryView
- let Some(metadata_field) = VariantArray::find_metadata_field(inner) else {
+ // Note the specification allows for any order so we must search by name
+
+ // Ensure the StructArray has a metadata field of BinaryView
+ let Some(metadata_field) = inner.column_by_name("metadata") else {
return Err(ArrowError::InvalidArgumentError(
"Invalid VariantArray: StructArray must contain a 'metadata' field".to_string(),
));
};
- if metadata_field.data_type() != &DataType::BinaryView {
+ let Some(metadata) = metadata_field.as_binary_view_opt() else {
return Err(ArrowError::NotYetImplemented(format!(
"VariantArray 'metadata' field must be BinaryView, got {}",
metadata_field.data_type()
)));
- }
- let Some(value_field) = VariantArray::find_value_field(inner) else {
- return Err(ArrowError::InvalidArgumentError(
- "Invalid VariantArray: StructArray must contain a 'value' field".to_string(),
- ));
};
- if value_field.data_type() != &DataType::BinaryView {
- return Err(ArrowError::NotYetImplemented(format!(
- "VariantArray 'value' field must be BinaryView, got {}",
- value_field.data_type()
- )));
- }
+
+ // Find the value field, if present
+ let value_field = inner.column_by_name("value");
+ let value = value_field
+ .map(|v| match v.as_binary_view_opt() {
+ Some(bv) => Ok(bv),
+ None => Err(ArrowError::NotYetImplemented(format!(
+ "VariantArray 'value' field must be BinaryView, got {}",
+ v.data_type()
+ ))),
+ })
+ .transpose()?;
+
+ // Find the typed_value field, if present
+ let typed_value = inner.column_by_name("typed_value");
+
+ // Note these clones are cheap, they just bump the ref count
+ let inner = inner.clone();
+ let metadata = metadata.clone();
+ let value = value.cloned();
+ let typed_value = typed_value.cloned();
+
+ let shredding_state = match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ },
+ (metadata, Some(value), None) => ShreddingState::Unshredded { metadata, value },
+ (metadata, None, Some(typed_value)) => ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ },
+ (_metadata_field, None, None) => {
+ return Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ )));
+ }
+ };
Review Comment:
Good idea -- done in 924155ca68
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -186,13 +340,11 @@ impl Array for VariantArray {
}
fn slice(&self, offset: usize, length: usize) -> ArrayRef {
- let slice = self.inner.slice(offset, length);
- let met = self.metadata_ref.slice(offset, length);
- let val = self.value_ref.slice(offset, length);
+ let inner = self.inner.slice(offset, length);
+ let shredding_state = self.shredding_state.slice(offset, length);
Arc::new(Self {
Review Comment:
We could definitely add a fast path for slicing the whole array.
I am not quite sure what ShreddingState::All and ShreddingState::Mixed is supposed to represent
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
I think you are probably right that the path handling should be handled elsewhere I haven't worked out exactly how extracting paths would be implemented. Maybe that would be a good follow on PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257483716
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
Review Comment:
yeah, I agree it is a matter of preference. If someone else feels strongly, I am not opposed to changing it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2256887779
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
Review Comment:
> Part of me wonders whether this "shredding state" enum is actually helping, vs. just storing `value` and `typed_value` as options?
The reason I first introduced the enum was that in `variant_get` I needed to dispatch to different methods based on the type of shredding, and I found it myself really wanting to have names for them rather than just the combinations of value and typed value.
Another reason reason was that I ended up with redundant error checking that was in the `VariantArray::try_new` (though if perfectly null arrays are allowed, maybe this becomes irrelevant)
> Especially since the [shredding spec](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding) seems to suggest that none-none case is a valid combination?
I personally think having names for the different shredding types makes working with the code easier, even if all 4 combinations are valid.
I think you are right, however, that the shredding spec allows both to be None. I double checked the [Arrow Proposal](https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?tab=t.0#heading=h.ru2zdn2szuc4) as it also implies both being null is permitted:
> An optional field named value that is binary, large_binary, or binary_view
> An optional field named typed_value which can be any primitive type or be a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257063486
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
Review Comment:
> The reason I first introduced the enum was that in `variant_get` I needed to dispatch to different methods based on the type of shredding, and I found it myself really wanting to have names for them rather than just the combinations of value and typed value.
I guess... but it seems like there's so much code just managing the resulting enum variants that I question whether it's a net win. Like the getters for metadata, value, etc. or the above code that shrinks from 25 lines to seven while handling more cases.
Clearly either way can be made correct, tho, so I guess it's a matter of preference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2254458822
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
This is a great idea -- I am working to pull it into its own cast_to_variant function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3155741609
> Ping or request a review when ready?
I think this is ready for review - it has not substantially changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2260889030
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ ))),
+ }
+ }
+
+ /// Return a reference to the metadata field
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ match self {
+ ShreddingState::Unshredded { metadata, .. } => metadata,
+ ShreddingState::FullyShredded { metadata, .. } => metadata,
+ ShreddingState::PartiallyShredded { metadata, .. } => metadata,
+ }
+ }
+
+ /// Return a reference to the value field, if present
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ match self {
+ ShreddingState::Unshredded { value, .. } => Some(value),
+ ShreddingState::FullyShredded { .. } => None,
+ ShreddingState::PartiallyShredded { value, .. } => Some(value),
+ }
+ }
+
+ /// Return a reference to the typed_value field, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ match self {
+ ShreddingState::Unshredded { .. } => None,
+ ShreddingState::FullyShredded { typed_value, .. } => Some(typed_value),
+ ShreddingState::PartiallyShredded { typed_value, .. } => Some(typed_value),
+ }
+ }
+
+ /// Slice all the underlying arrays
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ match self {
+ ShreddingState::Unshredded { metadata, value } => ShreddingState::Unshredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ },
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => ShreddingState::FullyShredded {
+ metadata: metadata.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => ShreddingState::PartiallyShredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ }
+ }
+}
+
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
Review Comment:
I couldn't make that work as it needs more information than just the native type (e.g. the precision/scale for decimals).
@superserious-dev 's `cast_conversion` macro I think can be adapted eventually to handle this case
https://github.com/apache/arrow-rs/pull/8074/files#diff-753713bd9a1945a5fc998fab6018eac0b08ebb4d04d3258b2b615498152a9303R46
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2249345215
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,171 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
Review Comment:
this one is a bit complicated due to generics but I think it will work for all primitive types (ints, floats, etc)
It handles extracting strongly typed values from variants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] WIP [Variant] Strawman `variant_get` for shredded variants [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2242462833
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -39,30 +40,117 @@ pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
)
})?;
+ match (
+ variant_array.metadata_field(),
+ variant_array.value_field(),
+ variant_array.typed_value_field(),
+ ) {
+ (metadata_field, Some(value_field), Some(typed_value_field)) => variant_get_shredded(
+ variant_array,
+ metadata_field,
+ value_field,
+ typed_value_field,
+ options,
+ ),
+ (metadata_field, Some(value_field), None) => {
+ variant_get_unshredded(variant_array, metadata_field, value_field, options)
+ }
+ (metadata_field, None, Some(typed_value_field)) => variant_get_perfectly_shredded(
+ variant_array,
+ metadata_field,
+ typed_value_field,
+ options,
+ ),
+ (metadata_field, None, None) => {
+ // This should not be possible, but return an erro just in case
+ return Err(ArrowError::InvalidArgumentError(format!(
+ "VariantArray has neither value nor typed_value field",
+ )));
+ }
+ }
+}
+
+/// variant_get for a shredded variant array
Review Comment:
I have thought a lot more about this last night and I have a plan -- will try and code it up later today / tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2255159393
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -93,35 +88,45 @@ impl VariantArray {
"Invalid VariantArray: requires StructArray as input".to_string(),
));
};
- // Ensure the StructArray has a metadata field of BinaryView
- let Some(metadata_field) = VariantArray::find_metadata_field(inner) else {
+ // Note the specification allows for any order so we must search by name
+
+ // Ensure the StructArray has a metadata field of BinaryView
+ let Some(metadata_field) = inner.column_by_name("metadata") else {
return Err(ArrowError::InvalidArgumentError(
"Invalid VariantArray: StructArray must contain a 'metadata' field".to_string(),
));
};
- if metadata_field.data_type() != &DataType::BinaryView {
+ let Some(metadata) = metadata_field.as_binary_view_opt() else {
return Err(ArrowError::NotYetImplemented(format!(
"VariantArray 'metadata' field must be BinaryView, got {}",
metadata_field.data_type()
)));
- }
- let Some(value_field) = VariantArray::find_value_field(inner) else {
- return Err(ArrowError::InvalidArgumentError(
- "Invalid VariantArray: StructArray must contain a 'value' field".to_string(),
- ));
};
- if value_field.data_type() != &DataType::BinaryView {
- return Err(ArrowError::NotYetImplemented(format!(
- "VariantArray 'value' field must be BinaryView, got {}",
- value_field.data_type()
- )));
- }
+
+ // Find the value field, if present
+ let value_field = inner.column_by_name("value");
+ let value = value_field
+ .map(|v| match v.as_binary_view_opt() {
+ Some(bv) => Ok(bv),
+ None => Err(ArrowError::NotYetImplemented(format!(
+ "VariantArray 'value' field must be BinaryView, got {}",
+ v.data_type()
+ ))),
+ })
+ .transpose()?;
Review Comment:
This goes from `Option` to `Option-Option` to `Option-Result` to `Result-Option` to `Option`? Phew!
nit: potential small simplification (`unwrap_or_else` actually useful):
```suggestion
let value = inner
.column_by_name("value")
.map(|v| v.as_binary_view_opt.unwrap_or_else(|| {
ArrowError::NotYetImplemented(format!(
"VariantArray 'value' field must be BinaryView, got {}",
v.data_type()
))
}))
.transpose()?;
```
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -93,35 +88,45 @@ impl VariantArray {
"Invalid VariantArray: requires StructArray as input".to_string(),
));
};
- // Ensure the StructArray has a metadata field of BinaryView
- let Some(metadata_field) = VariantArray::find_metadata_field(inner) else {
+ // Note the specification allows for any order so we must search by name
+
+ // Ensure the StructArray has a metadata field of BinaryView
+ let Some(metadata_field) = inner.column_by_name("metadata") else {
return Err(ArrowError::InvalidArgumentError(
"Invalid VariantArray: StructArray must contain a 'metadata' field".to_string(),
));
};
- if metadata_field.data_type() != &DataType::BinaryView {
+ let Some(metadata) = metadata_field.as_binary_view_opt() else {
return Err(ArrowError::NotYetImplemented(format!(
Review Comment:
aside: Out of curiosity, do we have any style/preference guidelines for
```rust
let Some(foo) = foo_opt else {
return Err(...);
};
```
vs.
```rust
let foo = foo_opt.unwrap_or_else(|| {
...
})?;
```
?
The former seems both more direct and less verbose, at least in this case?
Maybe `unwrap_or_else` is mostly (only?) useful as part of a bigger monadic chain?
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
Review Comment:
```suggestion
ShreddingState::FullyShredded { typed_value, .. } => {
```
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
Review Comment:
Part of me wonders whether this "shredding state" enum is actually helping, vs. just storing `value` and `typed_value` as options? Especially since the [shredding spec](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding) seems to suggest that none-none case is a valid combination?
> ```
> required group measurement (VARIANT) {
> required binary metadata;
> optional binary value;
> optional int64 typed_value;
> }
> ```
and
> <img width="658" height="84" alt="image" src="https://github.com/user-attachments/assets/b93e99c0-012a-4ecf-8580-245c3790d7f7" />
(the file should be able to omit both physical columns, in case of a perfectly shredded all-null the logical column)
```suggestion
// Always prefer typed_value over value (if present).
match self.typed_value.as_ref().and_then(|tv| typed_value_to_variant(tv, index)) {
Some(typed_value) => typed_value,
None => self.value.as_ref().map_or(
Variant::Null,
|v| Variant::new(self.metadata.value(index), v.value(index)),
)),
}
```
(this assumes that `typed_value_to_variant` includes the null check and returns `Option<Variant>`)
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
Review Comment:
These become trivial if we ditch the `ShreddingState` enum
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ ))),
+ }
+ }
+
+ /// Return a reference to the metadata field
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ match self {
+ ShreddingState::Unshredded { metadata, .. } => metadata,
+ ShreddingState::FullyShredded { metadata, .. } => metadata,
+ ShreddingState::PartiallyShredded { metadata, .. } => metadata,
+ }
+ }
+
+ /// Return a reference to the value field, if present
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ match self {
+ ShreddingState::Unshredded { value, .. } => Some(value),
+ ShreddingState::FullyShredded { .. } => None,
+ ShreddingState::PartiallyShredded { value, .. } => Some(value),
+ }
+ }
+
+ /// Return a reference to the typed_value field, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ match self {
+ ShreddingState::Unshredded { .. } => None,
+ ShreddingState::FullyShredded { typed_value, .. } => Some(typed_value),
+ ShreddingState::PartiallyShredded { typed_value, .. } => Some(typed_value),
+ }
+ }
+
+ /// Slice all the underlying arrays
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ match self {
+ ShreddingState::Unshredded { metadata, value } => ShreddingState::Unshredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ },
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => ShreddingState::FullyShredded {
+ metadata: metadata.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => ShreddingState::PartiallyShredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ }
+ }
+}
+
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
Review Comment:
Good use for [downcast_primitive_array](https://docs.rs/arrow/latest/arrow/macro.downcast_primitive_array.html) macro?
```suggestion
downcast_primitive_array! {
typed_value => Some(Variant::from(typed_value.value(index))),
```
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ ))),
+ }
+ }
+
+ /// Return a reference to the metadata field
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ match self {
+ ShreddingState::Unshredded { metadata, .. } => metadata,
+ ShreddingState::FullyShredded { metadata, .. } => metadata,
+ ShreddingState::PartiallyShredded { metadata, .. } => metadata,
+ }
+ }
+
+ /// Return a reference to the value field, if present
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ match self {
+ ShreddingState::Unshredded { value, .. } => Some(value),
+ ShreddingState::FullyShredded { .. } => None,
+ ShreddingState::PartiallyShredded { value, .. } => Some(value),
+ }
+ }
+
+ /// Return a reference to the typed_value field, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ match self {
+ ShreddingState::Unshredded { .. } => None,
+ ShreddingState::FullyShredded { typed_value, .. } => Some(typed_value),
+ ShreddingState::PartiallyShredded { typed_value, .. } => Some(typed_value),
+ }
+ }
+
+ /// Slice all the underlying arrays
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ match self {
+ ShreddingState::Unshredded { metadata, value } => ShreddingState::Unshredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ },
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => ShreddingState::FullyShredded {
+ metadata: metadata.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => ShreddingState::PartiallyShredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ }
+ }
+}
+
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
Review Comment:
Suggest to make this return `Option<Variant>` so callers don't have to check for null themselves.
```suggestion
fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
if typed_value.is_null(index) {
return None;
}
```
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
Review Comment:
I'm pretty sure the shredding spec allows this case. It corresponds to a perfectly shredded all-null column where the writer chose to omit both of them as unnecessary.
(see other comment above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2256847394
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -93,35 +88,45 @@ impl VariantArray {
"Invalid VariantArray: requires StructArray as input".to_string(),
));
};
- // Ensure the StructArray has a metadata field of BinaryView
- let Some(metadata_field) = VariantArray::find_metadata_field(inner) else {
+ // Note the specification allows for any order so we must search by name
+
+ // Ensure the StructArray has a metadata field of BinaryView
+ let Some(metadata_field) = inner.column_by_name("metadata") else {
return Err(ArrowError::InvalidArgumentError(
"Invalid VariantArray: StructArray must contain a 'metadata' field".to_string(),
));
};
- if metadata_field.data_type() != &DataType::BinaryView {
+ let Some(metadata) = metadata_field.as_binary_view_opt() else {
return Err(ArrowError::NotYetImplemented(format!(
Review Comment:
I personally prefer this pattern for the reasons you suggest. However, I don't know of anything formal we have discussed or agreed on
```rust
let Some(foo) = foo_opt else {
return Err(...);
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257095486
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
I had come up with some ideas before I remembered `downcast_to_primitive!` macro, but I think that would be the approach to take. Basically, a macro that just does the array casting and allows the caller to inject whatever they want. This site here would inject a scalar lookup, while the cast kernel would inject a for-loop.
Things get a lot trickier if we want to eliminate references to data types, tho, as the `downcast_to_primitive!` experiment in the other PR showed. We need a lot of homogeneity, or at least to know very clearly what data types are not covered so we can add specific match arms for them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261159145
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
> > We'd either have to do all the pathing here (recursively), or caller would have to to extract the struct/array once and recurse on its fields.
>
> Yes, I was imagining that this would do the pathing here on each row -- I am not sure what you mean by extract the struct array (as I don't think we can extract variants as structs in general, as the structures vary row by row)
Right, once we get into "true" (unshredded) variant data, we are stuck going row by row. But we can handle the pathing the the (possibly empty) prefix of perfectly shredded columns on a column-wise basis. Identify them when constructing the builders, and then just "snap" them into place when finalizing the builder. No per-row shenanigans needed. If a path isn't perfectly shredded, then the builder needs to take the variant column as input and finish the pathing row by row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261168165
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
For example, suppose that the user requested path `v:a.b.c.d::INT`, and that `a` and `b` were shredded but not `c`. Then the corresponding builder would take `v.typed_value.a.typed_value.b.value` as its input column, with `c.d::INT` as the path to extract.
Meanwhile, if the user also requested `v:a.b.x.y::INT` which happens to be perfectly shredded, then the corresponding "builder" would take `v.typed_value.a.typed_value.b.typed_value.x.typed_value.y.typed_value` as its input column and would not need to extract anything (so its per-row operation is a no-op) and its finalizer just needs to return the unchanged input `ArrayRef` so the parent builder for `v:a.b` can "snap" it into the correct place while assembling the `StructArray` that represents `v:a.b`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3165385000
Whops, I forgot to add some links. I will make a new PR for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2254732843
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
New PR
- https://github.com/apache/arrow-rs/pull/8044
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
Note I couldn't quite figure out how to reuse that kernel code in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3155189562
> > @scovich @carpecodeum and @Samyak2 this PR I think is ready for review
>
> Whoop, I somehow missed the notify. I'll try to take a look in the next day or two!
No worries -- I'll address @carpecodeum and @Samyak2 's comments in the mean time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2254722215
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +238,87 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let typed_value = typed_value.as_primitive::<Int32Type>();
+ Variant::from(typed_value.value(index))
+ }
+ // todo other types here
+ _ => {
+ todo!(); // Unsupported typed_value type
+ }
}
}
Review Comment:
I spent some time on this, and I found it was not as simple as I would have liked. Thus I have filed a ticket to track the idea
I filed a ticket here
- https://github.com/apache/arrow-rs/issues/8043
I will pull that code out into its own PR for easier review as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261124392
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
+ let variant = variant_array.value(i);
+ let Some(value) = T::from_variant(&variant) else {
+ if self.cast_options.safe {
+ // safe mode: append null if we can't convert
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ } else {
+ return Err(ArrowError::CastError(format!(
+ "Failed to extract primitive of type {} from variant {:?} at path {:?}",
+ self.as_type.data_type(),
+ variant,
+ self.path
+ )));
+ }
+ };
+
+ nulls.append_non_null();
+ values.push(value)
+ } else {
+ // otherwise we have a typed value, so we can use it directly
+ nulls.append_non_null();
+ values.push(typed_value.value(i));
+ }
+ }
+
+ let nulls = nulls.finish();
+ let array = PrimitiveArray::<T>::new(values.into(), nulls)
+ .with_data_type(self.as_type.data_type().clone());
+ Ok(Arc::new(array))
+ }
+
+ fn typed(
+ &self,
+ _variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // if the types match exactly, we can just return the typed_value
+ if typed_value.data_type() == self.as_type.data_type() {
+ Ok(typed_value.clone())
+ } else {
+ // TODO: try to cast the typed_value to the desired type?
Review Comment:
```suggestion
// TODO: try to cast the typed_value to the desired type?
// https://github.com/apache/arrow-rs/issues/8086
```
##########
parquet-variant-compute/src/variant_get/output/variant.rs:
##########
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::{VariantArray, VariantArrayBuilder};
+use arrow::array::{Array, ArrayRef, AsArray, BinaryViewArray};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, DataType};
+use parquet_variant::{Variant, VariantPath};
+use std::sync::Arc;
+
+/// Outputs VariantArrays
+pub(super) struct VariantOutputBuilder<'a> {
+ /// What path to extract
+ path: VariantPath<'a>,
+}
+
+impl<'a> VariantOutputBuilder<'a> {
+ pub(super) fn new(path: VariantPath<'a>) -> Self {
+ Self { path }
+ }
+}
+
+impl<'a> OutputBuilder for VariantOutputBuilder<'a> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ // TODO(perf): can reuse the metadata field here to avoid re-creating it
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // in this case dispatch on the typed_value and
+ // TODO macro'ize this using downcast! to handle all other primitive types
+ // TODO(perf): avoid builders entirely (and write the raw variant directly as we know the metadata is the same)
+ let mut array_builder = VariantArrayBuilder::new(variant_array.len());
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let primitive_array = typed_value.as_primitive::<Int32Type>();
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ array_builder.append_null();
+ continue;
+ }
+
+ if typed_value.is_null(i) {
+ // fall back to the value (variant) field
+ // (TODO could copy the variant bytes directly)
+ let value = variant_array.value(i);
+ array_builder.append_variant(value);
+ continue;
+ }
+
+ // otherwise we have a typed value, so we can use it directly
+ let int_value = primitive_array.value(i);
+ array_builder.append_variant(Variant::from(int_value));
+ }
+ }
+ dt => {
+ return Err(ArrowError::NotYetImplemented(format!(
+ "variant_get fully_shredded with typed_value={dt} is not implemented yet",
+ )));
+ }
+ };
+ Ok(Arc::new(array_builder.build()))
+ }
+
+ fn typed(
+ &self,
+ variant_array: &VariantArray,
+ // TODO(perf): can reuse the metadata field here to avoid re-creating it
+ _metadata: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // in this case dispatch on the typed_value and
+ // TODO macro'ize this using downcast! to handle all other primitive types
+ // TODO(perf): avoid builders entirely (and write the raw variant directly as we know the metadata is the same)
+ let mut array_builder = VariantArrayBuilder::new(variant_array.len());
+ match typed_value.data_type() {
+ DataType::Int32 => {
+ let primitive_array = typed_value.as_primitive::<Int32Type>();
+ for i in 0..variant_array.len() {
+ if primitive_array.is_null(i) {
+ array_builder.append_null();
+ continue;
+ }
+
+ let int_value = primitive_array.value(i);
+ array_builder.append_variant(Variant::from(int_value));
+ }
+ }
+ dt => {
+ return Err(ArrowError::NotYetImplemented(format!(
Review Comment:
- Filed https://github.com/apache/arrow-rs/issues/8086 for the remaining types
```suggestion
// https://github.com/apache/arrow-rs/issues/8086
return Err(ArrowError::NotYetImplemented(format!(
```
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
+ let variant = variant_array.value(i);
+ let Some(value) = T::from_variant(&variant) else {
+ if self.cast_options.safe {
+ // safe mode: append null if we can't convert
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ } else {
+ return Err(ArrowError::CastError(format!(
+ "Failed to extract primitive of type {} from variant {:?} at path {:?}",
+ self.as_type.data_type(),
+ variant,
+ self.path
+ )));
+ }
+ };
+
+ nulls.append_non_null();
+ values.push(value)
+ } else {
+ // otherwise we have a typed value, so we can use it directly
+ nulls.append_non_null();
+ values.push(typed_value.value(i));
+ }
+ }
+
+ let nulls = nulls.finish();
+ let array = PrimitiveArray::<T>::new(values.into(), nulls)
+ .with_data_type(self.as_type.data_type().clone());
+ Ok(Arc::new(array))
+ }
+
+ fn fully_shredded(
+ &self,
+ _variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // if the types match exactly, we can just return the typed_value
+ if typed_value.data_type() == self.as_type.data_type() {
+ Ok(typed_value.clone())
+ } else {
+ // TODO: try to cast the typed_value to the desired type?
+ Err(ArrowError::NotYetImplemented(format!(
Review Comment:
- filed https://github.com/apache/arrow-rs/issues/8086
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
- Tracking in https://github.com/apache/arrow-rs/issues/8083
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
Review Comment:
- https://github.com/apache/arrow-rs/issues/8088
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +139,217 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant<'_, '_> {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::Typed { typed_value, .. } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Represents the shredding state of a [`VariantArray`]
+///
+/// [`VariantArray`]s can be shredded according to the [Parquet Variant
+/// Shredding Spec]. Shredding means that the actual value is stored in a typed
+/// `typed_field` instead of the generic `value` field.
+///
+/// Both value and typed_value are optional fields used together to encode a
+/// single value. Values in the two fields must be interpreted according to the
+/// following table (see [Parquet Variant Shredding Spec] for more details):
+///
+/// | value | typed_value | Meaning |
+/// |----------|--------------|---------|
+/// | null | null | The value is missing; only valid for shredded object fields |
+/// | non-null | null | The value is present and may be any type, including `null` |
+/// | null | non-null | The value is present and is the shredded type |
+/// | non-null | non-null | The value is present and is a partially shredded object |
+///
+/// [Parquet Variant Shredding Spec]: https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding
+#[derive(Debug)]
+pub enum ShreddingState {
+ // TODO: add missing state where there is neither value nor typed_value
Review Comment:
```suggestion
// TODO: add missing state where there is neither value nor typed_value
// https://github.com/apache/arrow-rs/issues/8088
```
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +142,195 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded { typed_value, .. } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
Review Comment:
I am not sure
I filed a ticket to track adding AllNull:
- https://github.com/apache/arrow-rs/issues/8088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257590658
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ ))),
+ }
+ }
+
+ /// Return a reference to the metadata field
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ match self {
+ ShreddingState::Unshredded { metadata, .. } => metadata,
+ ShreddingState::FullyShredded { metadata, .. } => metadata,
+ ShreddingState::PartiallyShredded { metadata, .. } => metadata,
+ }
+ }
+
+ /// Return a reference to the value field, if present
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ match self {
+ ShreddingState::Unshredded { value, .. } => Some(value),
+ ShreddingState::FullyShredded { .. } => None,
+ ShreddingState::PartiallyShredded { value, .. } => Some(value),
+ }
+ }
+
+ /// Return a reference to the typed_value field, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ match self {
+ ShreddingState::Unshredded { .. } => None,
+ ShreddingState::FullyShredded { typed_value, .. } => Some(typed_value),
+ ShreddingState::PartiallyShredded { typed_value, .. } => Some(typed_value),
+ }
+ }
+
+ /// Slice all the underlying arrays
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ match self {
+ ShreddingState::Unshredded { metadata, value } => ShreddingState::Unshredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ },
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => ShreddingState::FullyShredded {
+ metadata: metadata.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => ShreddingState::PartiallyShredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ }
+ }
+}
+
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
Review Comment:
> For public API -- 100% agree. This is a private internal API tho, so it seems like we have room to do what we think makes the code simple/maintainable. Pulling important work inside the method instead of requiring all callers to remember it seems like a good example of that.
Sorry -- I wasn't clear -- the reason I was talking about `VariantArray::value` is that it is the only caller of `typed_value_to_variant` so if we return `Option` from this value, we would just be stuck at the same place 🤔
> Here, we're accessing is_null and value both inside the loop -- and the latter conditionally -- so I'd be very surprised if LLVM was willing to inject vectorization code that requires touching values the code said not to touch.
Yeah I agree I don't think it will make any difference for performance with Cariants. The primary reason in my mind is to be consistent with other APIs.
I think in an earlier version of this PR I actually had changed `value()` return `Option`. Maybe changing the signature is a good idea 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2257590658
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -135,36 +140,189 @@ impl VariantArray {
self.inner
}
+ /// Return the shredding state of this `VariantArray`
+ pub fn shredding_state(&self) -> &ShreddingState {
+ &self.shredding_state
+ }
+
/// Return the [`Variant`] instance stored at the given row
///
- /// Panics if the index is out of bounds.
+ /// Consistently with other Arrow arrays types, this API requires you to
+ /// check for nulls first using [`Self::is_valid`].
+ ///
+ /// # Panics
+ /// * if the index is out of bounds
+ /// * if the array value is null
+ ///
+ /// If this is a shredded variant but has no value at the shredded location, it
+ /// will return [`Variant::Null`].
+ ///
+ ///
+ /// # Performance Note
+ ///
+ /// This is certainly not the most efficient way to access values in a
+ /// `VariantArray`, but it is useful for testing and debugging.
///
/// Note: Does not do deep validation of the [`Variant`], so it is up to the
/// caller to ensure that the metadata and value were constructed correctly.
pub fn value(&self, index: usize) -> Variant {
- let metadata = self.metadata_field().as_binary_view().value(index);
- let value = self.value_field().as_binary_view().value(index);
- Variant::new(metadata, value)
+ match &self.shredding_state {
+ ShreddingState::Unshredded { metadata, value } => {
+ Variant::new(metadata.value(index), value.value(index))
+ }
+ ShreddingState::FullyShredded {
+ metadata: _,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::Null
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => {
+ if typed_value.is_null(index) {
+ Variant::new(metadata.value(index), value.value(index))
+ } else {
+ typed_value_to_variant(typed_value, index)
+ }
+ }
+ }
}
- fn find_metadata_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("metadata").cloned()
+ /// Return a reference to the metadata field of the [`StructArray`]
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ self.shredding_state.metadata_field()
}
- fn find_value_field(array: &StructArray) -> Option<ArrayRef> {
- array.column_by_name("value").cloned()
+ /// Return a reference to the value field of the `StructArray`
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ self.shredding_state.value_field()
}
- /// Return a reference to the metadata field of the [`StructArray`]
- pub fn metadata_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.metadata_ref
+ /// Return a reference to the typed_value field of the `StructArray`, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ self.shredding_state.typed_value_field()
}
+}
- /// Return a reference to the value field of the `StructArray`
- pub fn value_field(&self) -> &ArrayRef {
- // spec says fields order is not guaranteed, so we search by name
- &self.value_ref
+/// Variant arrays can be shredded in one of three states, encoded here
+#[derive(Debug)]
+pub enum ShreddingState {
+ /// This variant has no typed_value field
+ Unshredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ },
+ /// This variant has a typed_value field and no value field
+ /// meaning it is fully shredded (aka the value is stored in typed_value)
+ FullyShredded {
+ metadata: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+ /// This variant has both a value field and a typed_value field
+ /// meaning it is partially shredded: first the typed_value is used, and
+ /// if that is null, the value field is used.
+ PartiallyShredded {
+ metadata: BinaryViewArray,
+ value: BinaryViewArray,
+ typed_value: ArrayRef,
+ },
+}
+
+impl ShreddingState {
+ /// try to create a new `ShreddingState` from the given fields
+ pub fn try_new(
+ metadata: BinaryViewArray,
+ value: Option<BinaryViewArray>,
+ typed_value: Option<ArrayRef>,
+ ) -> Result<Self, ArrowError> {
+ match (metadata, value, typed_value) {
+ (metadata, Some(value), Some(typed_value)) => Ok(Self::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ }),
+ (metadata, Some(value), None) => Ok(Self::Unshredded { metadata, value }),
+ (metadata, None, Some(typed_value)) => Ok(Self::FullyShredded {
+ metadata,
+ typed_value,
+ }),
+ (_metadata_field, None, None) => Err(ArrowError::InvalidArgumentError(String::from(
+ "VariantArray has neither value nor typed_value field",
+ ))),
+ }
+ }
+
+ /// Return a reference to the metadata field
+ pub fn metadata_field(&self) -> &BinaryViewArray {
+ match self {
+ ShreddingState::Unshredded { metadata, .. } => metadata,
+ ShreddingState::FullyShredded { metadata, .. } => metadata,
+ ShreddingState::PartiallyShredded { metadata, .. } => metadata,
+ }
+ }
+
+ /// Return a reference to the value field, if present
+ pub fn value_field(&self) -> Option<&BinaryViewArray> {
+ match self {
+ ShreddingState::Unshredded { value, .. } => Some(value),
+ ShreddingState::FullyShredded { .. } => None,
+ ShreddingState::PartiallyShredded { value, .. } => Some(value),
+ }
+ }
+
+ /// Return a reference to the typed_value field, if present
+ pub fn typed_value_field(&self) -> Option<&ArrayRef> {
+ match self {
+ ShreddingState::Unshredded { .. } => None,
+ ShreddingState::FullyShredded { typed_value, .. } => Some(typed_value),
+ ShreddingState::PartiallyShredded { typed_value, .. } => Some(typed_value),
+ }
+ }
+
+ /// Slice all the underlying arrays
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ match self {
+ ShreddingState::Unshredded { metadata, value } => ShreddingState::Unshredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ },
+ ShreddingState::FullyShredded {
+ metadata,
+ typed_value,
+ } => ShreddingState::FullyShredded {
+ metadata: metadata.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ ShreddingState::PartiallyShredded {
+ metadata,
+ value,
+ typed_value,
+ } => ShreddingState::PartiallyShredded {
+ metadata: metadata.slice(offset, length),
+ value: value.slice(offset, length),
+ typed_value: typed_value.slice(offset, length),
+ },
+ }
+ }
+}
+
+/// returns the non-null element at index as a Variant
+fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant {
Review Comment:
> For public API -- 100% agree. This is a private internal API tho, so it seems like we have room to do what we think makes the code simple/maintainable. Pulling important work inside the method instead of requiring all callers to remember it seems like a good example of that.
Sorry -- I wasn't clear -- the reason I was talking about `VariantArray::value` is that it is the only caller of `typed_value_to_variant` so if we return `Option` from this value, we would just be stuck at the same place 🤔
> Here, we're accessing is_null and value both inside the loop -- and the latter conditionally -- so I'd be very surprised if LLVM was willing to inject vectorization code that requires touching values the code said not to touch.
Yeah I agree I don't think it will make any difference for performance. The primary reason in my mind is to be consistent with other APIs.
I think in an earlier version of this PR I actually had changed `value()` return `Option`. Maybe changing the signature is a good idea 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#issuecomment-3165065192
I have gone though the comments in this PR -- thank you @carpecodeum @Samyak2 and @scovich
I would very much like to unblock other people working on various parts of variant_get in parallel.
Thus, my plans now are:
1. Ensure the names for shredding align with the Variant shredding spec
2. Merge this PR in
3. File a bunch of follow on tickets -- several will be quite substantial like reading shredding objects, lists and pathing. However, I think we can work on much of it in parallel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261168165
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
For example, suppose that the user requested path `v:a.b.c.d::INT`, and that `a` and `b` were shredded but not `c`. Then the corresponding builder would take `v.typed_value.a.typed_value.b.value` as its input column, with `c.d::INT` as the path to extract on a per-row basis. And then its finalizer would return the resulting (now shredded) `ArrayRef` for the parent builder for `v:a.b` to snap into place.
Meanwhile, if the user also requested `v:a.b.x.y::INT` which happens to be perfectly shredded, then the corresponding "builder" would take `v.typed_value.a.typed_value.b.typed_value.x.typed_value.y.typed_value` as its input column. It would not need to extract anything (so its per-row operation is a no-op) and its finalizer just needs to return the unchanged input `ArrayRef` so the parent builder for `v:a.b` can "snap" it into the correct place while assembling the `StructArray` that represents `v:a.b`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Add `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261176877
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
Review Comment:
Sounds good -- I will be honest I didn't follow the entire thing but it sounds like it is on the right track to me
Given the length and size of this PR and the comments, I plan to merge it now and i think we'll end up working these questions out as we implement object shredding, under the aegis of
- https://github.com/apache/arrow-rs/issues/8083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261111449
##########
parquet-variant-compute/src/variant_get/output/primitive.rs:
##########
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::variant_get::output::OutputBuilder;
+use crate::VariantArray;
+use arrow::error::Result;
+
+use arrow::array::{
+ Array, ArrayRef, ArrowPrimitiveType, AsArray, BinaryViewArray, NullBufferBuilder,
+ PrimitiveArray,
+};
+use arrow::compute::{cast_with_options, CastOptions};
+use arrow::datatypes::Int32Type;
+use arrow_schema::{ArrowError, FieldRef};
+use parquet_variant::{Variant, VariantPath};
+use std::marker::PhantomData;
+use std::sync::Arc;
+
+/// Trait for Arrow primitive types that can be used in the output builder
+///
+/// This just exists to add a generic way to convert from Variant to the primitive type
+pub(super) trait ArrowPrimitiveVariant: ArrowPrimitiveType {
+ /// Try to extract the primitive value from a Variant, returning None if it
+ /// cannot be converted
+ ///
+ /// TODO: figure out how to handle coercion/casting
+ fn from_variant(variant: &Variant) -> Option<Self::Native>;
+}
+
+/// Outputs Primitive arrays
+pub(super) struct PrimitiveOutputBuilder<'a, T: ArrowPrimitiveVariant> {
+ /// What path to extract
+ path: VariantPath<'a>,
+ /// Returned output type
+ as_type: FieldRef,
+ /// Controls the casting behavior (e.g. error vs substituting null on cast error).
+ cast_options: CastOptions<'a>,
+ /// Phantom data for the primitive type
+ _phantom: PhantomData<T>,
+}
+
+impl<'a, T: ArrowPrimitiveVariant> PrimitiveOutputBuilder<'a, T> {
+ pub(super) fn new(
+ path: VariantPath<'a>,
+ as_type: FieldRef,
+ cast_options: CastOptions<'a>,
+ ) -> Self {
+ Self {
+ path,
+ as_type,
+ cast_options,
+ _phantom: PhantomData,
+ }
+ }
+}
+
+impl<'a, T: ArrowPrimitiveVariant> OutputBuilder for PrimitiveOutputBuilder<'a, T> {
+ fn partially_shredded(
+ &self,
+ variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // build up the output array element by element
+ let mut nulls = NullBufferBuilder::new(variant_array.len());
+ let mut values = Vec::with_capacity(variant_array.len());
+ let typed_value =
+ cast_with_options(typed_value, self.as_type.data_type(), &self.cast_options)?;
+ // downcast to the primitive array (e.g. Int32Array, Float64Array, etc)
+ let typed_value = typed_value.as_primitive::<T>();
+
+ for i in 0..variant_array.len() {
+ if variant_array.is_null(i) {
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ }
+
+ // if the typed value is null, decode the variant and extract the value
+ if typed_value.is_null(i) {
+ // todo follow path
+ let variant = variant_array.value(i);
+ let Some(value) = T::from_variant(&variant) else {
+ if self.cast_options.safe {
+ // safe mode: append null if we can't convert
+ nulls.append_null();
+ values.push(T::default_value()); // not used, placeholder
+ continue;
+ } else {
+ return Err(ArrowError::CastError(format!(
+ "Failed to extract primitive of type {} from variant {:?} at path {:?}",
+ self.as_type.data_type(),
+ variant,
+ self.path
+ )));
+ }
+ };
+
+ nulls.append_non_null();
+ values.push(value)
+ } else {
+ // otherwise we have a typed value, so we can use it directly
+ nulls.append_non_null();
+ values.push(typed_value.value(i));
+ }
+ }
+
+ let nulls = nulls.finish();
+ let array = PrimitiveArray::<T>::new(values.into(), nulls)
+ .with_data_type(self.as_type.data_type().clone());
+ Ok(Arc::new(array))
+ }
+
+ fn typed(
+ &self,
+ _variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ typed_value: &ArrayRef,
+ ) -> arrow::error::Result<ArrayRef> {
+ // if the types match exactly, we can just return the typed_value
+ if typed_value.data_type() == self.as_type.data_type() {
+ Ok(typed_value.clone())
+ } else {
+ // TODO: try to cast the typed_value to the desired type?
+ Err(ArrowError::NotYetImplemented(format!(
+ "variant_get fully_shredded as {:?} with typed_value={:?} is not implemented yet",
+ self.as_type.data_type(),
+ typed_value.data_type()
+ )))
+ }
+ }
+
+ fn unshredded(
+ &self,
+ _variant_array: &VariantArray,
+ _metadata: &BinaryViewArray,
+ _value_field: &BinaryViewArray,
+ ) -> Result<ArrayRef> {
+ Err(ArrowError::NotYetImplemented(String::from(
+ "variant_get unshredded to primitive types is not implemented yet",
+ )))
+ }
+}
+
+impl ArrowPrimitiveVariant for Int32Type {
+ fn from_variant(variant: &Variant) -> Option<Self::Native> {
+ variant.as_int32()
+ }
+}
+
+// todo for other primitive types
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Variant] Proposal for `variant_get` and Shredded `VariantArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8021:
URL: https://github.com/apache/arrow-rs/pull/8021#discussion_r2261105704
##########
parquet-variant-compute/src/variant_get/output/mod.rs:
##########
Review Comment:
Filed
- https://github.com/apache/arrow-rs/issues/8083
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org