You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "PinkCrow007 (via GitHub)" <gi...@apache.org> on 2025/04/11 00:15:08 UTC
[PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
PinkCrow007 opened a new pull request, #7404:
URL: https://github.com/apache/arrow-rs/pull/7404
This is a prototype that explores how we may implement Variant types in Parquet, including the structures, and as Arrow extension types. While this work is not yet completed, we (the CMU Variant team) are putting up this PR so everyone has a centralized place to communicate and coordinate work.
Currently, our implementation is set up as follows:
- In Arrow, we facilitate Variants using a Canonical Extension Type over binary types.
- In Parquet, we add Variant as a Logical Type.
- We also add a top-level "arrow-variant" to centralize parsing logic, similar to arrow-json.
Our current goal is to PR a minimum Variant with the intent of adding shredding functionality later. Our current roadmap is as follows:
- [x] Round-trip Variant between Arrow and Parquet while preserving data.
- [x] Implement Variant binary encoding/decoding.
- [x] Round-trip between JSON and Variant.
- [ ] Retrieve Variant data by key.
- [ ] Verify Variant binary compatibility with existing implementations.
I am the main engineer working on this project, with support from @mprammer, @pateljm, and @apavlo. This is our team's first Apache PR. We have been in contact with @alamb and @adriangb in the lead-up to this draft PR, and thank them both for their insight and support.
# Which issue does this PR close?
[[Parquet] Implement Variant type support in Parquet #6736](https://github.com/apache/arrow-rs/issues/6736)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "PinkCrow007 (via GitHub)" <gi...@apache.org>.
PinkCrow007 commented on code in PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#discussion_r2049303185
##########
arrow-array/src/array/variant_array.rs:
##########
@@ -0,0 +1,628 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::array::print_long_array;
+use crate::builder::{ArrayBuilder, BinaryBuilder};
+use crate::{Array, ArrayRef};
+use arrow_buffer::{Buffer, NullBuffer, OffsetBuffer, ScalarBuffer};
+use arrow_data::{ArrayData, ArrayDataBuilder};
+use arrow_schema::{ArrowError, DataType, Field};
+
+#[cfg(feature = "canonical_extension_types")]
+use arrow_schema::extension::Variant;
+use std::sync::Arc;
+use std::any::Any;
+
+/// An array of Variant values.
+///
+/// The Variant extension type stores data as two binary values: metadata and value.
+/// This array stores each Variant as a concatenated binary value (metadata + value).
+///
+/// # Example
+///
+/// ```
+/// use arrow_array::VariantArray;
+/// use arrow_schema::extension::Variant;
+/// use arrow_array::Array; // Import the Array trait
+///
+/// // Create metadata and value for each variant
+/// let metadata = vec![
+/// 0x01, // header: version=1, sorted=0, offset_size=1
+/// 0x01, // dictionary_size = 1
+/// 0x00, // offset 0
+/// 0x03, // offset 3
+/// b'k', b'e', b'y' // dictionary bytes
+/// ];
+/// let variant_type = Variant::new(metadata.clone(), vec![]);
+///
+/// // Create variants with different values
+/// let variants = vec![
+/// Variant::new(metadata.clone(), b"null".to_vec()),
+/// Variant::new(metadata.clone(), b"true".to_vec()),
+/// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+/// ];
+///
+/// // Create a VariantArray
+/// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+///
+/// // Access variants from the array
+/// assert_eq!(variant_array.len(), 3);
+/// let retrieved = variant_array.value(0).expect("Failed to get value");
+/// assert_eq!(retrieved.metadata(), &metadata);
+/// assert_eq!(retrieved.value(), b"null");
+/// ```
+#[cfg(feature = "canonical_extension_types")]
+pub mod variant_array_module {
+ use super::*;
+
+ /// An array of Variant values.
+ ///
+ /// The Variant extension type stores data as two binary values: metadata and value.
+ /// This array stores each Variant as a concatenated binary value (metadata + value).
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use arrow_array::VariantArray;
+ /// use arrow_schema::extension::Variant;
+ /// use arrow_array::Array; // Import the Array trait
+ ///
+ /// // Create metadata and value for each variant
+ /// let metadata = vec![
+ /// 0x01, // header: version=1, sorted=0, offset_size=1
+ /// 0x01, // dictionary_size = 1
+ /// 0x00, // offset 0
+ /// 0x03, // offset 3
+ /// b'k', b'e', b'y' // dictionary bytes
+ /// ];
+ /// let variant_type = Variant::new(metadata.clone(), vec![]);
+ ///
+ /// // Create variants with different values
+ /// let variants = vec![
+ /// Variant::new(metadata.clone(), b"null".to_vec()),
+ /// Variant::new(metadata.clone(), b"true".to_vec()),
+ /// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+ /// ];
+ ///
+ /// // Create a VariantArray
+ /// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+ ///
+ /// // Access variants from the array
+ /// assert_eq!(variant_array.len(), 3);
+ /// let retrieved = variant_array.value(0).expect("Failed to get value");
+ /// assert_eq!(retrieved.metadata(), &metadata);
+ /// assert_eq!(retrieved.value(), b"null");
+ /// ```
+ #[derive(Clone, Debug)]
+ pub struct VariantArray {
Review Comment:
> 5b564
Thanks @alamb ! Currently, the main branch of arrow-rs is using the same revision as I did, which doesn’t include the VARIANT logical type yet. I looked into updating to the latest revision from parquet-format, but it introduces new types like GEOMETRY, GEOGRAPHY, and other changes that are not yet supported in the arrow-rs main branch.
So if I stick with the current revision, I’ll need to manually add VARIANT to format.rs. If I upgrade to the latest parquet-format, I’ll also need to patch more places. Let me know if you think there’s a cleaner path forward!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "PinkCrow007 (via GitHub)" <gi...@apache.org>.
PinkCrow007 commented on code in PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#discussion_r2045042520
##########
arrow-schema/src/extension/canonical/variant.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Variant
+//!
+//! <https://arrow.apache.org/docs/format/CanonicalExtensions.html#variant>
+
+use base64::engine::Engine as _;
+use base64::engine::general_purpose::STANDARD;
+use crate::{extension::ExtensionType, ArrowError, DataType};
+
+/// The extension type for `Variant`.
+///
+/// Extension name: `arrow.variant`.
+///
+/// The storage type of this extension is **Binary or LargeBinary**.
+/// A Variant is a flexible structure that can store **Primitives, Arrays, or Objects**.
+/// It is stored as **two binary values**: `metadata` and `value`.
+///
+/// The **metadata field is required** and must be a valid Variant metadata string.
+/// The **value field is required** and contains the serialized Variant data.
+///
+/// <https://arrow.apache.org/docs/format/CanonicalExtensions.html#variant>
+#[derive(Debug, Clone, PartialEq)]
+pub struct Variant {
+ metadata: Vec<u8>, // Required binary metadata
+ value: Vec<u8>, // Required binary value
+}
+
+impl Variant {
+ /// Creates a new `Variant` with metadata and value.
+ pub fn new(metadata: Vec<u8>, value: Vec<u8>) -> Self {
+ Self { metadata, value }
+ }
+
+ /// Creates a Variant representing an empty structure.
+ pub fn empty() -> Result<Self, ArrowError> {
+ Err(ArrowError::InvalidArgumentError(
+ "Variant cannot be empty because metadata and value are required".to_owned(),
+ ))
+ }
+
+ /// Returns the metadata as a byte array.
+ pub fn metadata(&self) -> &[u8] {
+ &self.metadata
+ }
+
+ /// Returns the value as an byte array.
+ pub fn value(&self) -> &[u8] {
+ &self.value
+ }
+
+ /// Sets the value of the Variant.
+ pub fn set_value(mut self, value: Vec<u8>) -> Self {
+ self.value = value;
+ self
+ }
+}
+
+impl ExtensionType for Variant {
+ const NAME: &'static str = "arrow.variant";
+
+ type Metadata = Vec<u8>; // Metadata is directly Vec<u8>
+
+ fn metadata(&self) -> &Self::Metadata {
+ &self.metadata
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ Some(STANDARD.encode(&self.metadata)) // Encode metadata as STANDARD string
Review Comment:
Yeah, you're right. The ExtensionType metadata and the Variant metadata are not related. Right now the ExtensionType metadata isn’t storing anything meaningful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #7404: Variant Support for Arrow and Parquet [DRAFT]
URL: https://github.com/apache/arrow-rs/pull/7404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#discussion_r2039831129
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -4431,4 +4431,258 @@ mod tests {
assert_eq!(c0.len(), c1.len());
c0.iter().zip(c1.iter()).for_each(|(l, r)| assert_eq!(l, r));
}
+
+ #[test]
+ #[cfg(feature = "arrow_canonical_extension_types")]
Review Comment:
one of the benefits of using the existing StructArray mechanism rather than a new Array type is that we can likely reuse most of the existing parquet writing code, focusing on ensuring the metadata is written/read accurately
##########
arrow-schema/src/extension/canonical/variant.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Variant
+//!
+//! <https://arrow.apache.org/docs/format/CanonicalExtensions.html#variant>
+
+use base64::engine::Engine as _;
+use base64::engine::general_purpose::STANDARD;
+use crate::{extension::ExtensionType, ArrowError, DataType};
+
+/// The extension type for `Variant`.
+///
+/// Extension name: `arrow.variant`.
+///
+/// The storage type of this extension is **Binary or LargeBinary**.
+/// A Variant is a flexible structure that can store **Primitives, Arrays, or Objects**.
+/// It is stored as **two binary values**: `metadata` and `value`.
+///
+/// The **metadata field is required** and must be a valid Variant metadata string.
+/// The **value field is required** and contains the serialized Variant data.
+///
+/// <https://arrow.apache.org/docs/format/CanonicalExtensions.html#variant>
+#[derive(Debug, Clone, PartialEq)]
+pub struct Variant {
+ metadata: Vec<u8>, // Required binary metadata
+ value: Vec<u8>, // Required binary value
+}
+
+impl Variant {
+ /// Creates a new `Variant` with metadata and value.
+ pub fn new(metadata: Vec<u8>, value: Vec<u8>) -> Self {
+ Self { metadata, value }
+ }
+
+ /// Creates a Variant representing an empty structure.
+ pub fn empty() -> Result<Self, ArrowError> {
+ Err(ArrowError::InvalidArgumentError(
+ "Variant cannot be empty because metadata and value are required".to_owned(),
+ ))
+ }
+
+ /// Returns the metadata as a byte array.
+ pub fn metadata(&self) -> &[u8] {
+ &self.metadata
+ }
+
+ /// Returns the value as an byte array.
+ pub fn value(&self) -> &[u8] {
+ &self.value
+ }
+
+ /// Sets the value of the Variant.
+ pub fn set_value(mut self, value: Vec<u8>) -> Self {
+ self.value = value;
+ self
+ }
+}
+
+impl ExtensionType for Variant {
+ const NAME: &'static str = "arrow.variant";
+
+ type Metadata = Vec<u8>; // Metadata is directly Vec<u8>
+
+ fn metadata(&self) -> &Self::Metadata {
+ &self.metadata
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ Some(STANDARD.encode(&self.metadata)) // Encode metadata as STANDARD string
Review Comment:
My understanding of the extension metadata for a variant column isn't the same as the metadata for each variant value -- I probably don't fully understand the code but I think the extension type metadata is stored for *each field* where the variant's metadata is stored for each *row* (basically the word metadata is overloaded 😢 )
##########
arrow-schema/src/extension/canonical/variant.rs:
##########
@@ -0,0 +1,233 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Variant
+//!
+//! <https://arrow.apache.org/docs/format/CanonicalExtensions.html#variant>
+
+use base64::engine::Engine as _;
+use base64::engine::general_purpose::STANDARD;
+use crate::{extension::ExtensionType, ArrowError, DataType};
+
+/// The extension type for `Variant`.
+///
+/// Extension name: `arrow.variant`.
+///
+/// The storage type of this extension is **Binary or LargeBinary**.
+/// A Variant is a flexible structure that can store **Primitives, Arrays, or Objects**.
+/// It is stored as **two binary values**: `metadata` and `value`.
+///
+/// The **metadata field is required** and must be a valid Variant metadata string.
+/// The **value field is required** and contains the serialized Variant data.
+///
+/// <https://arrow.apache.org/docs/format/CanonicalExtensions.html#variant>
+#[derive(Debug, Clone, PartialEq)]
+pub struct Variant {
+ metadata: Vec<u8>, // Required binary metadata
Review Comment:
One concern I have with this approach is that it will *require* two allocations (and memory copies) to read a Variant value
I am hoping we can use Rust's borrow checker to do this with pointers (well, slices in rust) and no copying -- something like
```rust
pub struct Variant<'a> {
metadata: &'a [u8], // Required binary metadata
value: &'a [u8],
}
```
##########
arrow-array/src/array/variant_array.rs:
##########
@@ -0,0 +1,628 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::array::print_long_array;
+use crate::builder::{ArrayBuilder, BinaryBuilder};
+use crate::{Array, ArrayRef};
+use arrow_buffer::{Buffer, NullBuffer, OffsetBuffer, ScalarBuffer};
+use arrow_data::{ArrayData, ArrayDataBuilder};
+use arrow_schema::{ArrowError, DataType, Field};
+
+#[cfg(feature = "canonical_extension_types")]
+use arrow_schema::extension::Variant;
+use std::sync::Arc;
+use std::any::Any;
+
+/// An array of Variant values.
+///
+/// The Variant extension type stores data as two binary values: metadata and value.
+/// This array stores each Variant as a concatenated binary value (metadata + value).
+///
+/// # Example
+///
+/// ```
+/// use arrow_array::VariantArray;
+/// use arrow_schema::extension::Variant;
+/// use arrow_array::Array; // Import the Array trait
+///
+/// // Create metadata and value for each variant
+/// let metadata = vec![
+/// 0x01, // header: version=1, sorted=0, offset_size=1
+/// 0x01, // dictionary_size = 1
+/// 0x00, // offset 0
+/// 0x03, // offset 3
+/// b'k', b'e', b'y' // dictionary bytes
+/// ];
+/// let variant_type = Variant::new(metadata.clone(), vec![]);
+///
+/// // Create variants with different values
+/// let variants = vec![
+/// Variant::new(metadata.clone(), b"null".to_vec()),
+/// Variant::new(metadata.clone(), b"true".to_vec()),
+/// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+/// ];
+///
+/// // Create a VariantArray
+/// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+///
+/// // Access variants from the array
+/// assert_eq!(variant_array.len(), 3);
+/// let retrieved = variant_array.value(0).expect("Failed to get value");
+/// assert_eq!(retrieved.metadata(), &metadata);
+/// assert_eq!(retrieved.value(), b"null");
+/// ```
+#[cfg(feature = "canonical_extension_types")]
+pub mod variant_array_module {
+ use super::*;
+
+ /// An array of Variant values.
+ ///
+ /// The Variant extension type stores data as two binary values: metadata and value.
+ /// This array stores each Variant as a concatenated binary value (metadata + value).
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use arrow_array::VariantArray;
+ /// use arrow_schema::extension::Variant;
+ /// use arrow_array::Array; // Import the Array trait
+ ///
+ /// // Create metadata and value for each variant
+ /// let metadata = vec![
+ /// 0x01, // header: version=1, sorted=0, offset_size=1
+ /// 0x01, // dictionary_size = 1
+ /// 0x00, // offset 0
+ /// 0x03, // offset 3
+ /// b'k', b'e', b'y' // dictionary bytes
+ /// ];
+ /// let variant_type = Variant::new(metadata.clone(), vec![]);
+ ///
+ /// // Create variants with different values
+ /// let variants = vec![
+ /// Variant::new(metadata.clone(), b"null".to_vec()),
+ /// Variant::new(metadata.clone(), b"true".to_vec()),
+ /// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+ /// ];
+ ///
+ /// // Create a VariantArray
+ /// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+ ///
+ /// // Access variants from the array
+ /// assert_eq!(variant_array.len(), 3);
+ /// let retrieved = variant_array.value(0).expect("Failed to get value");
+ /// assert_eq!(retrieved.metadata(), &metadata);
+ /// assert_eq!(retrieved.value(), b"null");
+ /// ```
+ #[derive(Clone, Debug)]
+ pub struct VariantArray {
Review Comment:
One thing that we may want to explore is to model this as a `StructArray` with two fields, `"metadata"` and `"value"` which appears to be how @neilechao modeled it in the C++ implementation that merged a few days ago
- https://github.com/apache/arrow/pull/45375
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#discussion_r2049744027
##########
arrow-array/src/array/variant_array.rs:
##########
@@ -0,0 +1,628 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::array::print_long_array;
+use crate::builder::{ArrayBuilder, BinaryBuilder};
+use crate::{Array, ArrayRef};
+use arrow_buffer::{Buffer, NullBuffer, OffsetBuffer, ScalarBuffer};
+use arrow_data::{ArrayData, ArrayDataBuilder};
+use arrow_schema::{ArrowError, DataType, Field};
+
+#[cfg(feature = "canonical_extension_types")]
+use arrow_schema::extension::Variant;
+use std::sync::Arc;
+use std::any::Any;
+
+/// An array of Variant values.
+///
+/// The Variant extension type stores data as two binary values: metadata and value.
+/// This array stores each Variant as a concatenated binary value (metadata + value).
+///
+/// # Example
+///
+/// ```
+/// use arrow_array::VariantArray;
+/// use arrow_schema::extension::Variant;
+/// use arrow_array::Array; // Import the Array trait
+///
+/// // Create metadata and value for each variant
+/// let metadata = vec![
+/// 0x01, // header: version=1, sorted=0, offset_size=1
+/// 0x01, // dictionary_size = 1
+/// 0x00, // offset 0
+/// 0x03, // offset 3
+/// b'k', b'e', b'y' // dictionary bytes
+/// ];
+/// let variant_type = Variant::new(metadata.clone(), vec![]);
+///
+/// // Create variants with different values
+/// let variants = vec![
+/// Variant::new(metadata.clone(), b"null".to_vec()),
+/// Variant::new(metadata.clone(), b"true".to_vec()),
+/// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+/// ];
+///
+/// // Create a VariantArray
+/// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+///
+/// // Access variants from the array
+/// assert_eq!(variant_array.len(), 3);
+/// let retrieved = variant_array.value(0).expect("Failed to get value");
+/// assert_eq!(retrieved.metadata(), &metadata);
+/// assert_eq!(retrieved.value(), b"null");
+/// ```
+#[cfg(feature = "canonical_extension_types")]
+pub mod variant_array_module {
+ use super::*;
+
+ /// An array of Variant values.
+ ///
+ /// The Variant extension type stores data as two binary values: metadata and value.
+ /// This array stores each Variant as a concatenated binary value (metadata + value).
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use arrow_array::VariantArray;
+ /// use arrow_schema::extension::Variant;
+ /// use arrow_array::Array; // Import the Array trait
+ ///
+ /// // Create metadata and value for each variant
+ /// let metadata = vec![
+ /// 0x01, // header: version=1, sorted=0, offset_size=1
+ /// 0x01, // dictionary_size = 1
+ /// 0x00, // offset 0
+ /// 0x03, // offset 3
+ /// b'k', b'e', b'y' // dictionary bytes
+ /// ];
+ /// let variant_type = Variant::new(metadata.clone(), vec![]);
+ ///
+ /// // Create variants with different values
+ /// let variants = vec![
+ /// Variant::new(metadata.clone(), b"null".to_vec()),
+ /// Variant::new(metadata.clone(), b"true".to_vec()),
+ /// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+ /// ];
+ ///
+ /// // Create a VariantArray
+ /// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+ ///
+ /// // Access variants from the array
+ /// assert_eq!(variant_array.len(), 3);
+ /// let retrieved = variant_array.value(0).expect("Failed to get value");
+ /// assert_eq!(retrieved.metadata(), &metadata);
+ /// assert_eq!(retrieved.value(), b"null");
+ /// ```
+ #[derive(Clone, Debug)]
+ pub struct VariantArray {
Review Comment:
That PR is proof of concept to help get a change to the Parquet spec across the finish line. The format changes there, while including the Variant additions, also have not-yet-released bits. It's probably not worth waiting on those changes, and instead regen off of the last release version of parquet format (2.11.0), which appears to be commit `848302e179d7bb52a64caea6a058b3c08212787c`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#discussion_r2046627397
##########
arrow-array/src/array/variant_array.rs:
##########
@@ -0,0 +1,628 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::array::print_long_array;
+use crate::builder::{ArrayBuilder, BinaryBuilder};
+use crate::{Array, ArrayRef};
+use arrow_buffer::{Buffer, NullBuffer, OffsetBuffer, ScalarBuffer};
+use arrow_data::{ArrayData, ArrayDataBuilder};
+use arrow_schema::{ArrowError, DataType, Field};
+
+#[cfg(feature = "canonical_extension_types")]
+use arrow_schema::extension::Variant;
+use std::sync::Arc;
+use std::any::Any;
+
+/// An array of Variant values.
+///
+/// The Variant extension type stores data as two binary values: metadata and value.
+/// This array stores each Variant as a concatenated binary value (metadata + value).
+///
+/// # Example
+///
+/// ```
+/// use arrow_array::VariantArray;
+/// use arrow_schema::extension::Variant;
+/// use arrow_array::Array; // Import the Array trait
+///
+/// // Create metadata and value for each variant
+/// let metadata = vec![
+/// 0x01, // header: version=1, sorted=0, offset_size=1
+/// 0x01, // dictionary_size = 1
+/// 0x00, // offset 0
+/// 0x03, // offset 3
+/// b'k', b'e', b'y' // dictionary bytes
+/// ];
+/// let variant_type = Variant::new(metadata.clone(), vec![]);
+///
+/// // Create variants with different values
+/// let variants = vec![
+/// Variant::new(metadata.clone(), b"null".to_vec()),
+/// Variant::new(metadata.clone(), b"true".to_vec()),
+/// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+/// ];
+///
+/// // Create a VariantArray
+/// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+///
+/// // Access variants from the array
+/// assert_eq!(variant_array.len(), 3);
+/// let retrieved = variant_array.value(0).expect("Failed to get value");
+/// assert_eq!(retrieved.metadata(), &metadata);
+/// assert_eq!(retrieved.value(), b"null");
+/// ```
+#[cfg(feature = "canonical_extension_types")]
+pub mod variant_array_module {
+ use super::*;
+
+ /// An array of Variant values.
+ ///
+ /// The Variant extension type stores data as two binary values: metadata and value.
+ /// This array stores each Variant as a concatenated binary value (metadata + value).
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use arrow_array::VariantArray;
+ /// use arrow_schema::extension::Variant;
+ /// use arrow_array::Array; // Import the Array trait
+ ///
+ /// // Create metadata and value for each variant
+ /// let metadata = vec![
+ /// 0x01, // header: version=1, sorted=0, offset_size=1
+ /// 0x01, // dictionary_size = 1
+ /// 0x00, // offset 0
+ /// 0x03, // offset 3
+ /// b'k', b'e', b'y' // dictionary bytes
+ /// ];
+ /// let variant_type = Variant::new(metadata.clone(), vec![]);
+ ///
+ /// // Create variants with different values
+ /// let variants = vec![
+ /// Variant::new(metadata.clone(), b"null".to_vec()),
+ /// Variant::new(metadata.clone(), b"true".to_vec()),
+ /// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+ /// ];
+ ///
+ /// // Create a VariantArray
+ /// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+ ///
+ /// // Access variants from the array
+ /// assert_eq!(variant_array.len(), 3);
+ /// let retrieved = variant_array.value(0).expect("Failed to get value");
+ /// assert_eq!(retrieved.metadata(), &metadata);
+ /// assert_eq!(retrieved.value(), b"null");
+ /// ```
+ #[derive(Clone, Debug)]
+ pub struct VariantArray {
Review Comment:
> Make Variant an ExtensionType on top of Struct type in Arrow
> Use GroupType containing two binary fields in Parquet
> This aligns with the C++ version and avoids the conversion mismatch. What do you think — does that sound like a better design?
Yes I think this sounds right. The closer we can keep the arrow implementation to the spec and the other implementations the easier it will be to use in my opinion. If there is good reason to deviate we can discuss that too but in general I think following the existing implementations will make things easier
In terms of GroupType and encodings in Parquet, you may need to re-generate the thrift definitions to pick up the most recent version of the spec: https://github.com/apache/parquet-format/tree/master
We would do that by updating the revision here and then rerunning regen.sh:
https://github.com/apache/arrow-rs/blob/390de282d8c570ec9dd647efebd7a9b425a04d1f/parquet/regen.sh#L20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#discussion_r2049641522
##########
arrow-array/src/array/variant_array.rs:
##########
@@ -0,0 +1,628 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::array::print_long_array;
+use crate::builder::{ArrayBuilder, BinaryBuilder};
+use crate::{Array, ArrayRef};
+use arrow_buffer::{Buffer, NullBuffer, OffsetBuffer, ScalarBuffer};
+use arrow_data::{ArrayData, ArrayDataBuilder};
+use arrow_schema::{ArrowError, DataType, Field};
+
+#[cfg(feature = "canonical_extension_types")]
+use arrow_schema::extension::Variant;
+use std::sync::Arc;
+use std::any::Any;
+
+/// An array of Variant values.
+///
+/// The Variant extension type stores data as two binary values: metadata and value.
+/// This array stores each Variant as a concatenated binary value (metadata + value).
+///
+/// # Example
+///
+/// ```
+/// use arrow_array::VariantArray;
+/// use arrow_schema::extension::Variant;
+/// use arrow_array::Array; // Import the Array trait
+///
+/// // Create metadata and value for each variant
+/// let metadata = vec![
+/// 0x01, // header: version=1, sorted=0, offset_size=1
+/// 0x01, // dictionary_size = 1
+/// 0x00, // offset 0
+/// 0x03, // offset 3
+/// b'k', b'e', b'y' // dictionary bytes
+/// ];
+/// let variant_type = Variant::new(metadata.clone(), vec![]);
+///
+/// // Create variants with different values
+/// let variants = vec![
+/// Variant::new(metadata.clone(), b"null".to_vec()),
+/// Variant::new(metadata.clone(), b"true".to_vec()),
+/// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+/// ];
+///
+/// // Create a VariantArray
+/// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+///
+/// // Access variants from the array
+/// assert_eq!(variant_array.len(), 3);
+/// let retrieved = variant_array.value(0).expect("Failed to get value");
+/// assert_eq!(retrieved.metadata(), &metadata);
+/// assert_eq!(retrieved.value(), b"null");
+/// ```
+#[cfg(feature = "canonical_extension_types")]
+pub mod variant_array_module {
+ use super::*;
+
+ /// An array of Variant values.
+ ///
+ /// The Variant extension type stores data as two binary values: metadata and value.
+ /// This array stores each Variant as a concatenated binary value (metadata + value).
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use arrow_array::VariantArray;
+ /// use arrow_schema::extension::Variant;
+ /// use arrow_array::Array; // Import the Array trait
+ ///
+ /// // Create metadata and value for each variant
+ /// let metadata = vec![
+ /// 0x01, // header: version=1, sorted=0, offset_size=1
+ /// 0x01, // dictionary_size = 1
+ /// 0x00, // offset 0
+ /// 0x03, // offset 3
+ /// b'k', b'e', b'y' // dictionary bytes
+ /// ];
+ /// let variant_type = Variant::new(metadata.clone(), vec![]);
+ ///
+ /// // Create variants with different values
+ /// let variants = vec![
+ /// Variant::new(metadata.clone(), b"null".to_vec()),
+ /// Variant::new(metadata.clone(), b"true".to_vec()),
+ /// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+ /// ];
+ ///
+ /// // Create a VariantArray
+ /// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+ ///
+ /// // Access variants from the array
+ /// assert_eq!(variant_array.len(), 3);
+ /// let retrieved = variant_array.value(0).expect("Failed to get value");
+ /// assert_eq!(retrieved.metadata(), &metadata);
+ /// assert_eq!(retrieved.value(), b"null");
+ /// ```
+ #[derive(Clone, Debug)]
+ pub struct VariantArray {
Review Comment:
> I looked into updating to the latest revision from parquet-format, but it introduces new types like GEOMETRY, GEOGRAPHY, and other changes that are not yet supported in the arrow-rs main branch.
I think it is fine to add the type definitions for those new types.
@etseidl actually has a PR where he updated the thrift definition here: https://github.com/apache/arrow-rs/pull/7408/files#diff-d4b6c8a220629cfe5ac02a85cd537547b2e5c77b01e0793ced98adeb809bcde4R20
Though it is not clear what the status of that PR is.
I will try and find time to write up some tickets with some smaller sub tasks tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#discussion_r2047543959
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -4431,4 +4431,258 @@ mod tests {
assert_eq!(c0.len(), c1.len());
c0.iter().zip(c1.iter()).for_each(|(l, r)| assert_eq!(l, r));
}
+
+ #[test]
+ #[cfg(feature = "arrow_canonical_extension_types")]
Review Comment:
+1 for re-using struct array machinery.
It should also make it easier to do granular projection pushdown (only read data for a single field)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#issuecomment-2816034012
FYI I have started breaking the work down into smaller tickets -- here is the list: https://github.com/apache/arrow-rs/issues/6736#issuecomment-2816032074
- @PinkCrow007 perhaps you are interested in starting on https://github.com/apache/arrow-rs/issues/7424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "thinkharderdev (via GitHub)" <gi...@apache.org>.
thinkharderdev commented on code in PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#discussion_r2047155023
##########
arrow-array/src/array/variant_array.rs:
##########
@@ -0,0 +1,628 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::array::print_long_array;
+use crate::builder::{ArrayBuilder, BinaryBuilder};
+use crate::{Array, ArrayRef};
+use arrow_buffer::{Buffer, NullBuffer, OffsetBuffer, ScalarBuffer};
+use arrow_data::{ArrayData, ArrayDataBuilder};
+use arrow_schema::{ArrowError, DataType, Field};
+
+#[cfg(feature = "canonical_extension_types")]
+use arrow_schema::extension::Variant;
+use std::sync::Arc;
+use std::any::Any;
+
+/// An array of Variant values.
+///
+/// The Variant extension type stores data as two binary values: metadata and value.
+/// This array stores each Variant as a concatenated binary value (metadata + value).
+///
+/// # Example
+///
+/// ```
+/// use arrow_array::VariantArray;
+/// use arrow_schema::extension::Variant;
+/// use arrow_array::Array; // Import the Array trait
+///
+/// // Create metadata and value for each variant
+/// let metadata = vec![
+/// 0x01, // header: version=1, sorted=0, offset_size=1
+/// 0x01, // dictionary_size = 1
+/// 0x00, // offset 0
+/// 0x03, // offset 3
+/// b'k', b'e', b'y' // dictionary bytes
+/// ];
+/// let variant_type = Variant::new(metadata.clone(), vec![]);
+///
+/// // Create variants with different values
+/// let variants = vec![
+/// Variant::new(metadata.clone(), b"null".to_vec()),
+/// Variant::new(metadata.clone(), b"true".to_vec()),
+/// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+/// ];
+///
+/// // Create a VariantArray
+/// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+///
+/// // Access variants from the array
+/// assert_eq!(variant_array.len(), 3);
+/// let retrieved = variant_array.value(0).expect("Failed to get value");
+/// assert_eq!(retrieved.metadata(), &metadata);
+/// assert_eq!(retrieved.value(), b"null");
+/// ```
+#[cfg(feature = "canonical_extension_types")]
+pub mod variant_array_module {
+ use super::*;
+
+ /// An array of Variant values.
+ ///
+ /// The Variant extension type stores data as two binary values: metadata and value.
+ /// This array stores each Variant as a concatenated binary value (metadata + value).
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use arrow_array::VariantArray;
+ /// use arrow_schema::extension::Variant;
+ /// use arrow_array::Array; // Import the Array trait
+ ///
+ /// // Create metadata and value for each variant
+ /// let metadata = vec![
+ /// 0x01, // header: version=1, sorted=0, offset_size=1
+ /// 0x01, // dictionary_size = 1
+ /// 0x00, // offset 0
+ /// 0x03, // offset 3
+ /// b'k', b'e', b'y' // dictionary bytes
+ /// ];
+ /// let variant_type = Variant::new(metadata.clone(), vec![]);
+ ///
+ /// // Create variants with different values
+ /// let variants = vec![
+ /// Variant::new(metadata.clone(), b"null".to_vec()),
+ /// Variant::new(metadata.clone(), b"true".to_vec()),
+ /// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+ /// ];
+ ///
+ /// // Create a VariantArray
+ /// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+ ///
+ /// // Access variants from the array
+ /// assert_eq!(variant_array.len(), 3);
+ /// let retrieved = variant_array.value(0).expect("Failed to get value");
+ /// assert_eq!(retrieved.metadata(), &metadata);
+ /// assert_eq!(retrieved.value(), b"null");
+ /// ```
+ #[derive(Clone, Debug)]
+ pub struct VariantArray {
Review Comment:
Just to chime in here, I think this is the correct way to go as well. It allows readers to read the metadata without necessarily reading the payload
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "PinkCrow007 (via GitHub)" <gi...@apache.org>.
PinkCrow007 commented on PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#issuecomment-2822560319
Thanks everyone for the helpful suggestions! Just a quick update — I’ve adjusted the design to use ExtensionType over **Struct** in Arrow and **GroupType** in Parquet. Read/write Parquet, binary encoding, and JSON conversion all continue to work as expected after the change.
Hopefully this can serve as a reference going forward.
I’ll follow the smaller tickets Andrew outlined to continue the work. Thanks again everyone!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#issuecomment-2797109942
This is great thank you so much for putting up the PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#issuecomment-2892638496
There's a discussion happening on the Arrow dev mailing list about adding Variant as a canonical extension type in the Arrow spec. Input from the Arrow Rust developer community would be much appreciated.
- The discussion thread in the mailing list archive: https://lists.apache.org/thread/w06cxdojjcmry4m9vb0bo7owd1jsbtz5
- How to subscribe to the Arrow dev mailing list: https://arrow.apache.org/community/#mailing-lists
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#issuecomment-4805869092
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "aihuaxu (via GitHub)" <gi...@apache.org>.
aihuaxu commented on PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#issuecomment-2815958663
FYI. Variant logical type has been added to Parquet Java (see https://github.com/apache/parquet-java/pull/3072). The annotation will be `VARIANT(<specVersion>)` as mentioned in the last Parquet sync up. cc @alamb @PinkCrow007 in case
we are adding logical type in this PR.
> In Parquet, we add Variant as a Logical Type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#issuecomment-2809146066
Here are my planned next steps:
1. I will start writing up writing up some tickets
2. I will work on getting example binary data for variant (e.g. https://github.com/apache/parquet-testing/issues/75)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant Support for Arrow and Parquet [DRAFT] [arrow-rs]
Posted by "PinkCrow007 (via GitHub)" <gi...@apache.org>.
PinkCrow007 commented on code in PR #7404:
URL: https://github.com/apache/arrow-rs/pull/7404#discussion_r2043575052
##########
arrow-array/src/array/variant_array.rs:
##########
@@ -0,0 +1,628 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::array::print_long_array;
+use crate::builder::{ArrayBuilder, BinaryBuilder};
+use crate::{Array, ArrayRef};
+use arrow_buffer::{Buffer, NullBuffer, OffsetBuffer, ScalarBuffer};
+use arrow_data::{ArrayData, ArrayDataBuilder};
+use arrow_schema::{ArrowError, DataType, Field};
+
+#[cfg(feature = "canonical_extension_types")]
+use arrow_schema::extension::Variant;
+use std::sync::Arc;
+use std::any::Any;
+
+/// An array of Variant values.
+///
+/// The Variant extension type stores data as two binary values: metadata and value.
+/// This array stores each Variant as a concatenated binary value (metadata + value).
+///
+/// # Example
+///
+/// ```
+/// use arrow_array::VariantArray;
+/// use arrow_schema::extension::Variant;
+/// use arrow_array::Array; // Import the Array trait
+///
+/// // Create metadata and value for each variant
+/// let metadata = vec![
+/// 0x01, // header: version=1, sorted=0, offset_size=1
+/// 0x01, // dictionary_size = 1
+/// 0x00, // offset 0
+/// 0x03, // offset 3
+/// b'k', b'e', b'y' // dictionary bytes
+/// ];
+/// let variant_type = Variant::new(metadata.clone(), vec![]);
+///
+/// // Create variants with different values
+/// let variants = vec![
+/// Variant::new(metadata.clone(), b"null".to_vec()),
+/// Variant::new(metadata.clone(), b"true".to_vec()),
+/// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+/// ];
+///
+/// // Create a VariantArray
+/// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+///
+/// // Access variants from the array
+/// assert_eq!(variant_array.len(), 3);
+/// let retrieved = variant_array.value(0).expect("Failed to get value");
+/// assert_eq!(retrieved.metadata(), &metadata);
+/// assert_eq!(retrieved.value(), b"null");
+/// ```
+#[cfg(feature = "canonical_extension_types")]
+pub mod variant_array_module {
+ use super::*;
+
+ /// An array of Variant values.
+ ///
+ /// The Variant extension type stores data as two binary values: metadata and value.
+ /// This array stores each Variant as a concatenated binary value (metadata + value).
+ ///
+ /// # Example
+ ///
+ /// ```
+ /// use arrow_array::VariantArray;
+ /// use arrow_schema::extension::Variant;
+ /// use arrow_array::Array; // Import the Array trait
+ ///
+ /// // Create metadata and value for each variant
+ /// let metadata = vec![
+ /// 0x01, // header: version=1, sorted=0, offset_size=1
+ /// 0x01, // dictionary_size = 1
+ /// 0x00, // offset 0
+ /// 0x03, // offset 3
+ /// b'k', b'e', b'y' // dictionary bytes
+ /// ];
+ /// let variant_type = Variant::new(metadata.clone(), vec![]);
+ ///
+ /// // Create variants with different values
+ /// let variants = vec![
+ /// Variant::new(metadata.clone(), b"null".to_vec()),
+ /// Variant::new(metadata.clone(), b"true".to_vec()),
+ /// Variant::new(metadata.clone(), b"{\"a\": 1}".to_vec()),
+ /// ];
+ ///
+ /// // Create a VariantArray
+ /// let variant_array = VariantArray::from_variants(variant_type, variants.clone()).expect("Failed to create VariantArray");
+ ///
+ /// // Access variants from the array
+ /// assert_eq!(variant_array.len(), 3);
+ /// let retrieved = variant_array.value(0).expect("Failed to get value");
+ /// assert_eq!(retrieved.metadata(), &metadata);
+ /// assert_eq!(retrieved.value(), b"null");
+ /// ```
+ #[derive(Clone, Debug)]
+ pub struct VariantArray {
Review Comment:
Thanks so much @alamb! Your brilliant comment and the C++ implementation really got me thinking — maybe I should tackle this from the ground up.
Currently, in my implementation:
1. In Arrow: Variant is an ExtensionType over **Binary**
2. In Parquet: Variant is implemented as a **PrimitiveType** (not a GroupType)
- It’s a PrimitiveType where metadata and value are concatenated into one binary blob
- VariantArray handles the parsing/splitting of this binary data
I initially tried using GroupType but faced issues during arrow_to_parquet conversion. Since the Arrow side is a single Binary ExtensionType, mapping it to a GroupType (which contains two separate binary fields) caused a mismatch in column expectations that was tricky to resolve cleanly.
However, your comment and the C++ design raise an important question about alignment and extensibility. Would it be better to:
1. Make Variant an ExtensionType on top of **Struct** type in Arrow
2. Use **GroupType** containing two binary fields in Parquet
This aligns with the C++ version and avoids the conversion mismatch. What do you think — does that sound like a better design?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org