You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/09/16 20:27:41 UTC
[PR] Support reading/writing `VariantArray` to parquet, `VariantArray` does not implement `Array` [arrow-rs]
alamb opened a new pull request, #8365:
URL: https://github.com/apache/arrow-rs/pull/8365
# Which issue does this PR close?
- closes https://github.com/apache/arrow-rs/issues/8296
- closes https://github.com/apache/arrow-rs/issues/8319
# Rationale for this change
We need a way to write Variant encoded data to/from parquet, and the current
way the VariantArray is implemented doesn't work (panics when writing to parquet)
TODO:
- [ ] examples for going to/from Array --> VariantArray based on the Field's metadata
- [ ] Add the Field metadata based on the logical annotation from parquet
# What changes are included in this PR?
1. remove the `Array` impl for `VariantArray`, which forces explict conversions back/forth when writing
As a follow on PR I plan to give the same basic treatment to `ShreddedVariantStateArray`
# Are these changes tested?
yes, by existing tests and ndw tests
# Are there any user-facing changes?
Yes, but this is not yet stable / released, so these changes have no impact on the releasability of this code
1. `VariantArray` no longer implements `Array`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2356615886
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -617,59 +710,6 @@ fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant<'_, '
}
}
-impl Array for VariantArray {
Review Comment:
I can really remember what the problem was. I will try it again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2362510174
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -2295,7 +2295,7 @@ mod tests {
let batch = record_reader.next().unwrap().unwrap();
assert_eq!(batch.num_rows(), 1);
- let expected_schema = Schema::new(Fields::from(vec![Field::new(
+ let expected_schema = Schema::new(vec![Field::new(
Review Comment:
yes, it was something I found while cleaning up some code in this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2355969295
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -617,59 +710,6 @@ fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant<'_, '
}
}
-impl Array for VariantArray {
Review Comment:
It had something to do with references and not having the right ones or something -- I can't remember. Before merging this one I'll file a ticket for doing that too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2355939699
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -617,59 +710,6 @@ fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant<'_, '
}
}
-impl Array for VariantArray {
Review Comment:
Interesting. Was it somehow more complex than the top-level array? Or just a whole new set of knock-on effects to deal with? I thought it had a _very_ limited usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#issuecomment-3303566694
> the parquet reader will read Binary columns as Binary rather than BinaryView, which is somewhat awkward.
Should we consider just adding support for Binary columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Path Finding: Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2364659371
##########
parquet/src/variant.rs:
##########
@@ -108,6 +98,202 @@
//! ```
//!
//! # Example: Reading a Parquet file with Variant column
-//! (TODO: add example)
+//!
+//! Use the [`VariantType`] extension type to find the Variant column:
+//!
+//! ```
+//! # use std::sync::Arc;
+//! # use std::path::PathBuf;
+//! # use arrow_array::{ArrayRef, RecordBatch, RecordBatchReader};
+//! # use parquet::variant::{Variant, VariantArray, VariantType};
+//! # use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+//! # fn main() -> Result<(), parquet::errors::ParquetError> {
+//! # use arrow_array::StructArray;
+//! # fn file_path() -> PathBuf { // return a testing file path
+//! # PathBuf::from(arrow::util::test_util::parquet_test_data())
+//! # .join("..")
+//! # .join("shredded_variant")
+//! # .join("case-075.parquet")
+//! # }
+//! // Read the Parquet file using standard Arrow Parquet reader
+//! let file = std::fs::File::open(file_path())?;
+//! let mut reader = ArrowReaderBuilder::try_new(file)?.build()?;
+//!
+//! // You can find the Variant using the VariantType extension type
+//! let schema = reader.schema();
+//! let field = schema.field_with_name("var")?;
+//! assert!(field.try_extension_type::<VariantType>().is_ok());
+//!
+//! // The reader will yield RecordBatches with a StructArray
+//! // to convert them to VariantArray, use VariantArray::try_new
+//! let batch = reader.next().unwrap().unwrap();
+//! let col = batch.column_by_name("var").unwrap();
+//! let var_array = VariantArray::try_new(&col)?;
+//! assert_eq!(var_array.len(), 1);
+//! let var_value: Variant = var_array.value(0);
+//! assert_eq!(var_value, Variant::from("iceberg")); // the value in case-075.parquet
+//! # Ok(())
+//! # }
+//! ```
pub use parquet_variant::*;
-pub use parquet_variant_compute as compute;
+pub use parquet_variant_compute::*;
+
+#[cfg(test)]
+mod tests {
+ use crate::arrow::arrow_reader::ArrowReaderBuilder;
+ use crate::arrow::ArrowWriter;
+ use crate::file::metadata::{ParquetMetaData, ParquetMetaDataReader};
+ use crate::file::reader::ChunkReader;
+ use arrow::util::test_util::parquet_test_data;
+ use arrow_array::{ArrayRef, RecordBatch};
+ use arrow_schema::Schema;
+ use bytes::Bytes;
+ use parquet_variant::{Variant, VariantBuilderExt};
+ use parquet_variant_compute::{VariantArray, VariantArrayBuilder, VariantType};
+ use std::path::PathBuf;
+ use std::sync::Arc;
+
+ #[test]
+ fn roundtrip_basic() {
+ roundtrip(variant_array());
+ }
+
+ /// Ensure a file with Variant LogicalType, written by another writer in
+ /// parquet-testing, can be read as a VariantArray
+ #[test]
+ fn read_logical_type() {
Review Comment:
Or, just move the additional assertions to the doc test, hidden behind `#` if users don't care to see them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Path Finding: Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2365619934
##########
parquet/src/variant.rs:
##########
@@ -108,6 +98,202 @@
//! ```
//!
//! # Example: Reading a Parquet file with Variant column
-//! (TODO: add example)
+//!
+//! Use the [`VariantType`] extension type to find the Variant column:
+//!
+//! ```
+//! # use std::sync::Arc;
+//! # use std::path::PathBuf;
+//! # use arrow_array::{ArrayRef, RecordBatch, RecordBatchReader};
+//! # use parquet::variant::{Variant, VariantArray, VariantType};
+//! # use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+//! # fn main() -> Result<(), parquet::errors::ParquetError> {
+//! # use arrow_array::StructArray;
+//! # fn file_path() -> PathBuf { // return a testing file path
+//! # PathBuf::from(arrow::util::test_util::parquet_test_data())
+//! # .join("..")
+//! # .join("shredded_variant")
+//! # .join("case-075.parquet")
+//! # }
+//! // Read the Parquet file using standard Arrow Parquet reader
+//! let file = std::fs::File::open(file_path())?;
+//! let mut reader = ArrowReaderBuilder::try_new(file)?.build()?;
+//!
+//! // You can find the Variant using the VariantType extension type
+//! let schema = reader.schema();
+//! let field = schema.field_with_name("var")?;
+//! assert!(field.try_extension_type::<VariantType>().is_ok());
+//!
+//! // The reader will yield RecordBatches with a StructArray
+//! // to convert them to VariantArray, use VariantArray::try_new
+//! let batch = reader.next().unwrap().unwrap();
+//! let col = batch.column_by_name("var").unwrap();
+//! let var_array = VariantArray::try_new(&col)?;
+//! assert_eq!(var_array.len(), 1);
+//! let var_value: Variant = var_array.value(0);
+//! assert_eq!(var_value, Variant::from("iceberg")); // the value in case-075.parquet
+//! # Ok(())
+//! # }
+//! ```
pub use parquet_variant::*;
-pub use parquet_variant_compute as compute;
+pub use parquet_variant_compute::*;
+
+#[cfg(test)]
+mod tests {
+ use crate::arrow::arrow_reader::ArrowReaderBuilder;
+ use crate::arrow::ArrowWriter;
+ use crate::file::metadata::{ParquetMetaData, ParquetMetaDataReader};
+ use crate::file::reader::ChunkReader;
+ use arrow::util::test_util::parquet_test_data;
+ use arrow_array::{ArrayRef, RecordBatch};
+ use arrow_schema::Schema;
+ use bytes::Bytes;
+ use parquet_variant::{Variant, VariantBuilderExt};
+ use parquet_variant_compute::{VariantArray, VariantArrayBuilder, VariantType};
+ use std::path::PathBuf;
+ use std::sync::Arc;
+
+ #[test]
+ fn roundtrip_basic() {
+ roundtrip(variant_array());
+ }
+
+ /// Ensure a file with Variant LogicalType, written by another writer in
+ /// parquet-testing, can be read as a VariantArray
+ #[test]
+ fn read_logical_type() {
Review Comment:
that is a great idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2360909097
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -24,12 +24,54 @@ use arrow::datatypes::{
Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
UInt32Type, UInt64Type, UInt8Type,
};
+use arrow_schema::extension::ExtensionType;
use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Uuid;
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
+/// Variant Canonical Extension Type
+pub struct VariantType;
+
+impl ExtensionType for VariantType {
+ const NAME: &'static str = "parquet.variant";
+
+ // Variants have no extension metadata
+ type Metadata = ();
+
+ fn metadata(&self) -> &Self::Metadata {
+ &()
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ None
+ }
+
+ fn deserialize_metadata(_metadata: Option<&str>) -> Result<Self::Metadata, ArrowError> {
+ Ok(())
+ }
+
+ fn supports_data_type(&self, data_type: &DataType) -> Result<(), ArrowError> {
+ // Note don't check for metadata/value fields here because they may be
+ // absent in shredded variants
+ if matches!(data_type, DataType::Struct(_)) {
+ Ok(())
Review Comment:
The scenario I'm thinking of is a more forgiving replacement for normal json parsing, where the requested schema has a mix of strongly-typed and variant fields. So parse JSON to variant, and then use `variant_get` to extract the actual schema of interest. Fields that are known to have inconsistent (or legally flexible) behavior would be kept as variant, with strict typing enforcement applied only to the strongly typed fields (similar to today's JSON parsing with a schema). But in that case, we have a (potentially deeply) nested struct with multiple variant leaf fields, and fetching individual fields one at a time would be pretty annoying.
If the caller _wants_ a specific schema -- whether binary variant or a specific shredding -- it's easy enough to pass that. The difficulty comes if the caller just wants to get back whatever flavor of variant is already there (without shredding or unshredding it first).
I don't see how `cast_to_variant` would help in that case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2363897588
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -87,20 +88,20 @@ pub(crate) fn follow_shredded_path_element<'a>(
return Ok(missing_path_step());
};
- let field = field
- .as_any()
- .downcast_ref::<ShreddedVariantFieldArray>()
- .ok_or_else(|| {
- // TODO: Should we blow up? Or just end the traversal and let the normal
- // variant pathing code sort out the mess that it must anyway be
- // prepared to handle?
- ArrowError::InvalidArgumentError(format!(
- "Expected a ShreddedVariantFieldArray, got {:?} instead",
- field.data_type(),
- ))
- })?;
-
- Ok(ShreddedPathStep::Success(field.shredding_state()))
+ let struct_array = field.as_struct_opt().ok_or_else(|| {
+ // TODO: Should we blow up? Or just end the traversal and let the normal
+ // variant pathing code sort out the mess that it must anyway be
+ // prepared to handle?
+ ArrowError::InvalidArgumentError(format!(
+ "Expected Struct array while following path, got {}",
+ field.data_type(),
+ ))
+ })?;
+
+ let shredding_state =
+ ShreddedVariantFieldArray::shredding_state_from_struct_array(struct_array)?;
Review Comment:
This is a great idea. I did it in https://github.com/apache/arrow-rs/pull/8392 (commit 4b1daab75f)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2363797842
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -1651,7 +1633,7 @@ mod test {
}
}
- Arc::new(builder.build())
+ builder.build().into()
Review Comment:
Fixed in https://github.com/apache/arrow-rs/pull/8392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2362524999
##########
parquet/src/variant.rs:
##########
@@ -108,6 +98,202 @@
//! ```
//!
//! # Example: Reading a Parquet file with Variant column
-//! (TODO: add example)
+//!
+//! Use the [`VariantType`] extension type to find the Variant column:
+//!
+//! ```
+//! # use std::sync::Arc;
+//! # use std::path::PathBuf;
+//! # use arrow_array::{ArrayRef, RecordBatch, RecordBatchReader};
+//! # use parquet::variant::{Variant, VariantArray, VariantType};
+//! # use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+//! # fn main() -> Result<(), parquet::errors::ParquetError> {
+//! # use arrow_array::StructArray;
+//! # fn file_path() -> PathBuf { // return a testing file path
+//! # PathBuf::from(arrow::util::test_util::parquet_test_data())
+//! # .join("..")
+//! # .join("shredded_variant")
+//! # .join("case-075.parquet")
+//! # }
+//! // Read the Parquet file using standard Arrow Parquet reader
+//! let file = std::fs::File::open(file_path())?;
+//! let mut reader = ArrowReaderBuilder::try_new(file)?.build()?;
+//!
+//! // You can find the Variant using the VariantType extension type
+//! let schema = reader.schema();
+//! let field = schema.field_with_name("var")?;
+//! assert!(field.try_extension_type::<VariantType>().is_ok());
+//!
+//! // The reader will yield RecordBatches with a StructArray
+//! // to convert them to VariantArray, use VariantArray::try_new
+//! let batch = reader.next().unwrap().unwrap();
+//! let col = batch.column_by_name("var").unwrap();
+//! let var_array = VariantArray::try_new(&col)?;
+//! assert_eq!(var_array.len(), 1);
+//! let var_value: Variant = var_array.value(0);
+//! assert_eq!(var_value, Variant::from("iceberg")); // the value in case-075.parquet
+//! # Ok(())
+//! # }
+//! ```
pub use parquet_variant::*;
-pub use parquet_variant_compute as compute;
+pub use parquet_variant_compute::*;
+
+#[cfg(test)]
+mod tests {
+ use crate::arrow::arrow_reader::ArrowReaderBuilder;
+ use crate::arrow::ArrowWriter;
+ use crate::file::metadata::{ParquetMetaData, ParquetMetaDataReader};
+ use crate::file::reader::ChunkReader;
+ use arrow::util::test_util::parquet_test_data;
+ use arrow_array::{ArrayRef, RecordBatch};
+ use arrow_schema::Schema;
+ use bytes::Bytes;
+ use parquet_variant::{Variant, VariantBuilderExt};
+ use parquet_variant_compute::{VariantArray, VariantArrayBuilder, VariantType};
+ use std::path::PathBuf;
+ use std::sync::Arc;
+
+ #[test]
+ fn roundtrip_basic() {
+ roundtrip(variant_array());
+ }
+
+ /// Ensure a file with Variant LogicalType, written by another writer in
+ /// parquet-testing, can be read as a VariantArray
+ #[test]
+ fn read_logical_type() {
Review Comment:
Yes, you are right -- it is the same. I wrote this test before the doctest. I'll remove this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2363801740
##########
parquet/src/variant.rs:
##########
@@ -108,6 +98,202 @@
//! ```
//!
//! # Example: Reading a Parquet file with Variant column
-//! (TODO: add example)
+//!
+//! Use the [`VariantType`] extension type to find the Variant column:
+//!
+//! ```
+//! # use std::sync::Arc;
+//! # use std::path::PathBuf;
+//! # use arrow_array::{ArrayRef, RecordBatch, RecordBatchReader};
+//! # use parquet::variant::{Variant, VariantArray, VariantType};
+//! # use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+//! # fn main() -> Result<(), parquet::errors::ParquetError> {
+//! # use arrow_array::StructArray;
+//! # fn file_path() -> PathBuf { // return a testing file path
+//! # PathBuf::from(arrow::util::test_util::parquet_test_data())
+//! # .join("..")
+//! # .join("shredded_variant")
+//! # .join("case-075.parquet")
+//! # }
+//! // Read the Parquet file using standard Arrow Parquet reader
+//! let file = std::fs::File::open(file_path())?;
+//! let mut reader = ArrowReaderBuilder::try_new(file)?.build()?;
+//!
+//! // You can find the Variant using the VariantType extension type
+//! let schema = reader.schema();
+//! let field = schema.field_with_name("var")?;
+//! assert!(field.try_extension_type::<VariantType>().is_ok());
+//!
+//! // The reader will yield RecordBatches with a StructArray
+//! // to convert them to VariantArray, use VariantArray::try_new
+//! let batch = reader.next().unwrap().unwrap();
+//! let col = batch.column_by_name("var").unwrap();
+//! let var_array = VariantArray::try_new(&col)?;
+//! assert_eq!(var_array.len(), 1);
+//! let var_value: Variant = var_array.value(0);
+//! assert_eq!(var_value, Variant::from("iceberg")); // the value in case-075.parquet
+//! # Ok(())
+//! # }
+//! ```
pub use parquet_variant::*;
-pub use parquet_variant_compute as compute;
+pub use parquet_variant_compute::*;
+
+#[cfg(test)]
+mod tests {
+ use crate::arrow::arrow_reader::ArrowReaderBuilder;
+ use crate::arrow::ArrowWriter;
+ use crate::file::metadata::{ParquetMetaData, ParquetMetaDataReader};
+ use crate::file::reader::ChunkReader;
+ use arrow::util::test_util::parquet_test_data;
+ use arrow_array::{ArrayRef, RecordBatch};
+ use arrow_schema::Schema;
+ use bytes::Bytes;
+ use parquet_variant::{Variant, VariantBuilderExt};
+ use parquet_variant_compute::{VariantArray, VariantArrayBuilder, VariantType};
+ use std::path::PathBuf;
+ use std::sync::Arc;
+
+ #[test]
+ fn roundtrip_basic() {
+ roundtrip(variant_array());
+ }
+
+ /// Ensure a file with Variant LogicalType, written by another writer in
+ /// parquet-testing, can be read as a VariantArray
+ #[test]
+ fn read_logical_type() {
Review Comment:
I double checked, and this test also does some more stuff like testing sizes. Do you think it is worth removing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2355594747
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -24,12 +24,54 @@ use arrow::datatypes::{
Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
UInt32Type, UInt64Type, UInt8Type,
};
+use arrow_schema::extension::ExtensionType;
use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Uuid;
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
+/// Variant Canonical Extension Type
+pub struct VariantType;
+
+impl ExtensionType for VariantType {
+ const NAME: &'static str = "parquet.variant";
+
+ // Variants have no extension metadata
+ type Metadata = ();
+
+ fn metadata(&self) -> &Self::Metadata {
+ &()
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ None
+ }
+
+ fn deserialize_metadata(_metadata: Option<&str>) -> Result<Self::Metadata, ArrowError> {
+ Ok(())
+ }
+
+ fn supports_data_type(&self, data_type: &DataType) -> Result<(), ArrowError> {
+ // Note don't check for metadata/value fields here because they may be
+ // absent in shredded variants
+ if matches!(data_type, DataType::Struct(_)) {
+ Ok(())
Review Comment:
One thing I thought of while working on shredding, variant_get, etc -- what if we supported a metadata that indicated the desired handling of shredded columns? So for example (naming TBD), a caller could annotate a binary variant type with "keep-shredding" to indicate that reads and variant_get calls should preserve the existing shredding structure of the underlying data, or "unshred" to indicate that such calls should return binary variant even if the underlying data are shredded. The output data would not contain any annotation, because it is what it is.
Such an approach would solve the problem of how to request partially shredded results using `variant_get`, for example. Today, we can get that behavior by passing `as_type=None`, but that doesn't work if I want a nested type with some fields shredded and others left as-is.
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -246,6 +288,57 @@ impl VariantArray {
pub fn typed_value_field(&self) -> Option<&ArrayRef> {
self.shredding_state.typed_value_field()
}
+
+ // TODO return a field with the correct metadata?
+ pub fn data_type(&self) -> &DataType {
+ self.inner.data_type()
+ }
+
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ let inner = self.inner.slice(offset, length);
+ let metadata = self.metadata.slice(offset, length);
+ let shredding_state = self.shredding_state.slice(offset, length);
+ Self {
+ inner,
+ metadata,
+ shredding_state,
+ }
+ }
+
+ pub fn len(&self) -> usize {
+ self.inner.len()
+ }
+
+ pub fn is_empty(&self) -> bool {
+ self.inner.is_empty()
+ }
+
+ pub fn nulls(&self) -> Option<&NullBuffer> {
+ self.inner.nulls()
+ }
+
+ /// Is the element at index null?
+ pub fn is_null(&self, index: usize) -> bool {
+ // TODO: should we account for shredded nulls here?
Review Comment:
Do you mean, return true here if `Variant::Null` in the `value` column and `NULL` in the `typed_value`?
I think that's more a job for `logical_nulls`?
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -617,59 +710,6 @@ fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant<'_, '
}
}
-impl Array for VariantArray {
Review Comment:
I think we also need to remove the `impl Array` for the shredded object field array as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Path Finding: Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#issuecomment-3321249270
The second (and final) PR resulting from this one is ready for review:
- https://github.com/apache/arrow-rs/pull/8408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#issuecomment-3304650255
I need to do a little more cleanup tomorrow morning, then this should be ready to go
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2361187972
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -24,12 +24,54 @@ use arrow::datatypes::{
Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
UInt32Type, UInt64Type, UInt8Type,
};
+use arrow_schema::extension::ExtensionType;
use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Uuid;
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
+/// Variant Canonical Extension Type
+pub struct VariantType;
+
+impl ExtensionType for VariantType {
+ const NAME: &'static str = "parquet.variant";
+
+ // Variants have no extension metadata
+ type Metadata = ();
+
+ fn metadata(&self) -> &Self::Metadata {
+ &()
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ None
+ }
+
+ fn deserialize_metadata(_metadata: Option<&str>) -> Result<Self::Metadata, ArrowError> {
+ Ok(())
+ }
+
+ fn supports_data_type(&self, data_type: &DataType) -> Result<(), ArrowError> {
+ // Note don't check for metadata/value fields here because they may be
+ // absent in shredded variants
+ if matches!(data_type, DataType::Struct(_)) {
+ Ok(())
Review Comment:
> I don't see how cast_to_variant would help in that case?
I don't think it would (I misunderstood what you were getting at)
> But in that case, we have a (potentially deeply) nested struct with multiple variant leaf fields, and fetching individual fields one at a time would be pretty annoying.
> The difficulty comes if the caller just wants to get back whatever flavor of variant is already there (without shredding or unshredding it first).
I am confused -- if we are parsing JSON, then it will always be variant (no shredded), so I am not sure how there would be different flavors of variant. Maybe this is a good thing for a call or some examples
Also I wonder if maybe we should open a new ticket as this conversation is likely to get pretty lost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2361172252
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -617,59 +710,6 @@ fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant<'_, '
}
}
-impl Array for VariantArray {
Review Comment:
I tried again and got it to work, more later
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -617,59 +710,6 @@ fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant<'_, '
}
}
-impl Array for VariantArray {
Review Comment:
I tried again and got it to work, am polishing now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#issuecomment-3303560469
> Nice. Initial pass didn't turn up anything scary?
Not really -- it seems to be going fine. The biggest issue I hit ss that currently the parquet reader will read Binary columns as Binary rather than BinaryView, which is somewhat awkward.
In this PR I still need to implement writing logical types too, and then clean up the code. I hope to have time to finish it later today or tomorrow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2362520661
##########
parquet/src/arrow/schema/extension.rs:
##########
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Arrow Extension Type Support for Parquet
+//!
+//! This module contains mapping code to map Parquet [`LogicalType`]s to/from
+//! Arrow [`ExtensionType`]s.
+//!
+//! Extension types are represented using the metadata from Arrow [`Field`]s
+//! with the key "ARROW:extension:name".
+
+use crate::basic::LogicalType;
+use crate::schema::types::Type;
+use arrow_schema::extension::ExtensionType;
+use arrow_schema::Field;
+
+/// Adds extension type metadata, if necessary, based on the Parquet field's
+/// [`LogicalType`]
+///
+/// Some Parquet logical types, such as Variant, do not map directly to an
+/// Arrow DataType, and instead are represented by an Arrow ExtensionType.
+/// Extension types are attached to Arrow Fields via metadata.
+pub(crate) fn add_extension_type(arrow_field: Field, parquet_type: &Type) -> Field {
+ let result = match parquet_type.get_basic_info().logical_type() {
+ #[cfg(feature = "variant_experimental")]
+ Some(LogicalType::Variant) => {
+ arrow_field.with_extension_type(parquet_variant_compute::VariantType)
+ }
+ // TODO add other LogicalTypes here
+ _ => arrow_field,
+ };
+ result
+}
+
+/// Return the Parquet logical type to use for the specified Arrow field, if any.
+#[cfg(feature = "variant_experimental")]
+pub(crate) fn logical_type_for_struct(field: &Field) -> Option<LogicalType> {
+ use parquet_variant_compute::VariantType;
+ if field.extension_type_name()? == VariantType::NAME {
+ return Some(LogicalType::Variant);
+ };
+ None
Review Comment:
May rationale here (which I will add as a comment) was that creating an `ArrowError` is expensive (as it allocates a String) so doing `try_extension_type().ok()` would add an allocation to all code paths even those that didn't have Variant type.
I will try and make this clearer, and maybe add some more docs to the extension type code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2363815639
##########
parquet/src/variant.rs:
##########
@@ -108,6 +98,202 @@
//! ```
//!
//! # Example: Reading a Parquet file with Variant column
-//! (TODO: add example)
+//!
+//! Use the [`VariantType`] extension type to find the Variant column:
+//!
+//! ```
+//! # use std::sync::Arc;
+//! # use std::path::PathBuf;
+//! # use arrow_array::{ArrayRef, RecordBatch, RecordBatchReader};
+//! # use parquet::variant::{Variant, VariantArray, VariantType};
+//! # use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+//! # fn main() -> Result<(), parquet::errors::ParquetError> {
+//! # use arrow_array::StructArray;
+//! # fn file_path() -> PathBuf { // return a testing file path
+//! # PathBuf::from(arrow::util::test_util::parquet_test_data())
+//! # .join("..")
+//! # .join("shredded_variant")
+//! # .join("case-075.parquet")
+//! # }
+//! // Read the Parquet file using standard Arrow Parquet reader
+//! let file = std::fs::File::open(file_path())?;
+//! let mut reader = ArrowReaderBuilder::try_new(file)?.build()?;
+//!
+//! // You can find the Variant using the VariantType extension type
Review Comment:
clarified in e2004751a5d5cc624c11ad6dd02a7ac3b64ec40f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2363799256
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -242,19 +243,15 @@ fn shredded_get_path(
/// quickly become annoying (and inefficient) to call `variant_get` for each leaf value in a struct or
/// list and then try to assemble the results.
pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
- let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
- ArrowError::InvalidArgumentError(
- "expected a VariantArray as the input for variant_get".to_owned(),
- )
- })?;
+ let variant_array = VariantArray::try_new(&input)?;
Review Comment:
Yes, excellent eyes. Fixed in https://github.com/apache/arrow-rs/pull/8392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Path Finding: Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed pull request #8365: Path Finding: Support reading/writing `VariantArray` to parquet with `Variant` LogicalType
URL: https://github.com/apache/arrow-rs/pull/8365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2356610808
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -24,12 +24,54 @@ use arrow::datatypes::{
Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
UInt32Type, UInt64Type, UInt8Type,
};
+use arrow_schema::extension::ExtensionType;
use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Uuid;
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
+/// Variant Canonical Extension Type
+pub struct VariantType;
+
+impl ExtensionType for VariantType {
+ const NAME: &'static str = "parquet.variant";
+
+ // Variants have no extension metadata
+ type Metadata = ();
+
+ fn metadata(&self) -> &Self::Metadata {
+ &()
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ None
+ }
+
+ fn deserialize_metadata(_metadata: Option<&str>) -> Result<Self::Metadata, ArrowError> {
+ Ok(())
+ }
+
+ fn supports_data_type(&self, data_type: &DataType) -> Result<(), ArrowError> {
+ // Note don't check for metadata/value fields here because they may be
+ // absent in shredded variants
+ if matches!(data_type, DataType::Struct(_)) {
+ Ok(())
Review Comment:
I don't fully understand the problem, and thus probably don't fully understand your proposal
I would expect that `variant_get` would always preserve the existing shredding structure, and if the user wants to unshred the result (e.g. ensure the output of variant_get is an unshredded variant) and that I would specify any conversions as part of `variant_get` rather than some annotation on the VariantArray itself
Maybe we could add a `variant_unshred` kernel, potentially with a path argument to only partially unshred an object 🤔 I am a little unclear on the usecase for that type of operation thoug
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -24,12 +24,54 @@ use arrow::datatypes::{
Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
UInt32Type, UInt64Type, UInt8Type,
};
+use arrow_schema::extension::ExtensionType;
use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Uuid;
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
+/// Variant Canonical Extension Type
+pub struct VariantType;
+
+impl ExtensionType for VariantType {
+ const NAME: &'static str = "parquet.variant";
+
+ // Variants have no extension metadata
+ type Metadata = ();
+
+ fn metadata(&self) -> &Self::Metadata {
+ &()
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ None
+ }
+
+ fn deserialize_metadata(_metadata: Option<&str>) -> Result<Self::Metadata, ArrowError> {
+ Ok(())
+ }
+
+ fn supports_data_type(&self, data_type: &DataType) -> Result<(), ArrowError> {
+ // Note don't check for metadata/value fields here because they may be
+ // absent in shredded variants
+ if matches!(data_type, DataType::Struct(_)) {
+ Ok(())
Review Comment:
I don't fully understand the problem, and thus probably don't fully understand your proposal
I would expect that `variant_get` would always preserve the existing shredding structure, and if the user wants to unshred the result (e.g. ensure the output of variant_get is an unshredded variant) and that I would specify any conversions as part of `variant_get` rather than some annotation on the VariantArray itself
Maybe we could add a `variant_unshred` kernel, potentially with a path argument to only partially unshred an object 🤔 I am a little unclear on the usecase for that type of operation though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2356687243
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -24,12 +24,54 @@ use arrow::datatypes::{
Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
UInt32Type, UInt64Type, UInt8Type,
};
+use arrow_schema::extension::ExtensionType;
use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Uuid;
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
+/// Variant Canonical Extension Type
+pub struct VariantType;
+
+impl ExtensionType for VariantType {
+ const NAME: &'static str = "parquet.variant";
+
+ // Variants have no extension metadata
+ type Metadata = ();
+
+ fn metadata(&self) -> &Self::Metadata {
+ &()
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ None
+ }
+
+ fn deserialize_metadata(_metadata: Option<&str>) -> Result<Self::Metadata, ArrowError> {
+ Ok(())
+ }
+
+ fn supports_data_type(&self, data_type: &DataType) -> Result<(), ArrowError> {
+ // Note don't check for metadata/value fields here because they may be
+ // absent in shredded variants
+ if matches!(data_type, DataType::Struct(_)) {
+ Ok(())
Review Comment:
There are a few cases:
* Pass `as_type=None` -- this would preserve existing shredding structure, if any
* Pass `as_type=Some(... VariantType...)` -- what schema should the underlying struct have, to:
* Force shredding as a specific schema
* Force unshredding to binary
* Return whatever was there (equivalent to passing None)
* Pass `as_type=Some(StructType(... VariantType ...))` -- this requests to shred the variant value as a struct, but that the field(s) inside that struct are variant-typed (not shredded). The same three questions apply, but now `as_type=None` is not a fallback
* We had talked previously about a providing function that could return the shredding schema for a given path, so the user can stub in the variant-typed fields however they want. But that's pretty fiddly.
* The alternative (which I proposed above) was to allow annotating the variant type to indicate that `variant_get` should preserve the underlying shredding structure, without having to actually specify it. Without that annotation, the variant's schema is taken literally and may cause shredding, reshredding, or unshredding depending on the underlying data's structure.
* Much cleaner, and actually simpler for the implementation as well. Once the field's path is exhausted, just return whatever was there (similar to how `as_type=None` is implemented today); if we passed an explicit shredding schema instead, then the implementation would have to examine every node of the schema, all the way to leaf level, in case the provided shredding structure doesn't actually match the underlying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2356687243
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -24,12 +24,54 @@ use arrow::datatypes::{
Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
UInt32Type, UInt64Type, UInt8Type,
};
+use arrow_schema::extension::ExtensionType;
use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Uuid;
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
+/// Variant Canonical Extension Type
+pub struct VariantType;
+
+impl ExtensionType for VariantType {
+ const NAME: &'static str = "parquet.variant";
+
+ // Variants have no extension metadata
+ type Metadata = ();
+
+ fn metadata(&self) -> &Self::Metadata {
+ &()
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ None
+ }
+
+ fn deserialize_metadata(_metadata: Option<&str>) -> Result<Self::Metadata, ArrowError> {
+ Ok(())
+ }
+
+ fn supports_data_type(&self, data_type: &DataType) -> Result<(), ArrowError> {
+ // Note don't check for metadata/value fields here because they may be
+ // absent in shredded variants
+ if matches!(data_type, DataType::Struct(_)) {
+ Ok(())
Review Comment:
There are a few cases:
* Pass `as_type=None` -- this would preserve existing shredding structure, if any
* Pass `as_type=Some(... VariantType...)` -- what schema should the underlying struct have, to:
* Force shredding as a specific schema
* Force unshredding to binary
* Return whatever was there (equivalent to passing None)
* Pass `as_type=Some(StructType(... VariantType ...))` -- this requests to shred the variant value as a struct, but that the field(s) inside that struct are variant-typed (not shredded). The same three questions apply, but now `as_type=None` is not a fallback
* We had talked previously about a providing function that could return the shredding schema for a given path, so the user can stub in the variant-typed fields however they want. But that's pretty fiddly.
* The alternative (which I proposed above) was to allow annotating the variant type to indicate that `variant_get` should preserve the underlying shredding structure, without having to actually specify it.
* Much cleaner, and actually simpler for the implementation as well. Once the field's path is exhausted, just return whatever was there (similar to how `as_type=None` is implemented today); if we passed an explicit shredding schema instead, then the implementation would have to examine every node of the schema, all the way to leaf level, in case the provided shredding structure doesn't actually match the underlying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2363815639
##########
parquet/src/variant.rs:
##########
@@ -108,6 +98,202 @@
//! ```
//!
//! # Example: Reading a Parquet file with Variant column
-//! (TODO: add example)
+//!
+//! Use the [`VariantType`] extension type to find the Variant column:
+//!
+//! ```
+//! # use std::sync::Arc;
+//! # use std::path::PathBuf;
+//! # use arrow_array::{ArrayRef, RecordBatch, RecordBatchReader};
+//! # use parquet::variant::{Variant, VariantArray, VariantType};
+//! # use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+//! # fn main() -> Result<(), parquet::errors::ParquetError> {
+//! # use arrow_array::StructArray;
+//! # fn file_path() -> PathBuf { // return a testing file path
+//! # PathBuf::from(arrow::util::test_util::parquet_test_data())
+//! # .join("..")
+//! # .join("shredded_variant")
+//! # .join("case-075.parquet")
+//! # }
+//! // Read the Parquet file using standard Arrow Parquet reader
+//! let file = std::fs::File::open(file_path())?;
+//! let mut reader = ArrowReaderBuilder::try_new(file)?.build()?;
+//!
+//! // You can find the Variant using the VariantType extension type
Review Comment:
clarified in https://github.com/apache/arrow-rs/pull/8365/commits/0bea1375dbf8ace69f724679239d9773968041af
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#issuecomment-3309715050
Update here is I think this PR is now in pretty good shape (it does the Logical type annotations and has examples of reading/writing them)
However, it is pretty large to request a review on. My plan is to break out the "remove Array impl` part into a separate PR for easier review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2362174150
##########
parquet/src/arrow/schema/extension.rs:
##########
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Arrow Extension Type Support for Parquet
+//!
+//! This module contains mapping code to map Parquet [`LogicalType`]s to/from
+//! Arrow [`ExtensionType`]s.
+//!
+//! Extension types are represented using the metadata from Arrow [`Field`]s
+//! with the key "ARROW:extension:name".
+
+use crate::basic::LogicalType;
+use crate::schema::types::Type;
+use arrow_schema::extension::ExtensionType;
+use arrow_schema::Field;
+
+/// Adds extension type metadata, if necessary, based on the Parquet field's
+/// [`LogicalType`]
+///
+/// Some Parquet logical types, such as Variant, do not map directly to an
+/// Arrow DataType, and instead are represented by an Arrow ExtensionType.
+/// Extension types are attached to Arrow Fields via metadata.
+pub(crate) fn add_extension_type(arrow_field: Field, parquet_type: &Type) -> Field {
+ let result = match parquet_type.get_basic_info().logical_type() {
+ #[cfg(feature = "variant_experimental")]
+ Some(LogicalType::Variant) => {
+ arrow_field.with_extension_type(parquet_variant_compute::VariantType)
+ }
+ // TODO add other LogicalTypes here
Review Comment:
Out of curiosity, how do the other parquet logical types map to arrow types? Seems like JSON, BSON, UUID, etc will face the same problem as Variant?
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -242,19 +243,15 @@ fn shredded_get_path(
/// quickly become annoying (and inefficient) to call `variant_get` for each leaf value in a struct or
/// list and then try to assemble the results.
pub fn variant_get(input: &ArrayRef, options: GetOptions) -> Result<ArrayRef> {
- let variant_array: &VariantArray = input.as_any().downcast_ref().ok_or_else(|| {
- ArrowError::InvalidArgumentError(
- "expected a VariantArray as the input for variant_get".to_owned(),
- )
- })?;
+ let variant_array = VariantArray::try_new(&input)?;
Review Comment:
Isn't `&input` a double-ref?
##########
parquet/src/arrow/schema/extension.rs:
##########
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Arrow Extension Type Support for Parquet
+//!
+//! This module contains mapping code to map Parquet [`LogicalType`]s to/from
+//! Arrow [`ExtensionType`]s.
+//!
+//! Extension types are represented using the metadata from Arrow [`Field`]s
+//! with the key "ARROW:extension:name".
+
+use crate::basic::LogicalType;
+use crate::schema::types::Type;
+use arrow_schema::extension::ExtensionType;
+use arrow_schema::Field;
+
+/// Adds extension type metadata, if necessary, based on the Parquet field's
+/// [`LogicalType`]
+///
+/// Some Parquet logical types, such as Variant, do not map directly to an
+/// Arrow DataType, and instead are represented by an Arrow ExtensionType.
+/// Extension types are attached to Arrow Fields via metadata.
+pub(crate) fn add_extension_type(arrow_field: Field, parquet_type: &Type) -> Field {
+ let result = match parquet_type.get_basic_info().logical_type() {
+ #[cfg(feature = "variant_experimental")]
+ Some(LogicalType::Variant) => {
+ arrow_field.with_extension_type(parquet_variant_compute::VariantType)
+ }
+ // TODO add other LogicalTypes here
+ _ => arrow_field,
+ };
+ result
+}
+
+/// Return the Parquet logical type to use for the specified Arrow field, if any.
+#[cfg(feature = "variant_experimental")]
+pub(crate) fn logical_type_for_struct(field: &Field) -> Option<LogicalType> {
+ use parquet_variant_compute::VariantType;
+ if field.extension_type_name()? == VariantType::NAME {
+ return Some(LogicalType::Variant);
+ };
+ None
Review Comment:
But shouldn't we follow the example in other parts of the code, and `try_extension_type` rather than merely checking the name? Otherwise, somebody could slap the variant annotation on some completely irrelevant struct, with no validation to catch the problem.
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -1651,7 +1633,7 @@ mod test {
}
}
- Arc::new(builder.build())
+ builder.build().into()
Review Comment:
aside: ironically, `into` would have worked with the original code as well, but I had chosen `Arc::new` as more readable.
But what decides between `into` (here) vs. `ArrayRef::from` (below)?
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -2295,7 +2295,7 @@ mod tests {
let batch = record_reader.next().unwrap().unwrap();
assert_eq!(batch.num_rows(), 1);
- let expected_schema = Schema::new(Fields::from(vec![Field::new(
+ let expected_schema = Schema::new(vec![Field::new(
Review Comment:
I guess `Fields::from` was never needed?
(seems unrelated to the VariantArray changes?)
##########
parquet/src/variant.rs:
##########
@@ -108,6 +98,202 @@
//! ```
//!
//! # Example: Reading a Parquet file with Variant column
-//! (TODO: add example)
+//!
+//! Use the [`VariantType`] extension type to find the Variant column:
+//!
+//! ```
+//! # use std::sync::Arc;
+//! # use std::path::PathBuf;
+//! # use arrow_array::{ArrayRef, RecordBatch, RecordBatchReader};
+//! # use parquet::variant::{Variant, VariantArray, VariantType};
+//! # use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+//! # fn main() -> Result<(), parquet::errors::ParquetError> {
+//! # use arrow_array::StructArray;
+//! # fn file_path() -> PathBuf { // return a testing file path
+//! # PathBuf::from(arrow::util::test_util::parquet_test_data())
+//! # .join("..")
+//! # .join("shredded_variant")
+//! # .join("case-075.parquet")
+//! # }
+//! // Read the Parquet file using standard Arrow Parquet reader
+//! let file = std::fs::File::open(file_path())?;
+//! let mut reader = ArrowReaderBuilder::try_new(file)?.build()?;
+//!
+//! // You can find the Variant using the VariantType extension type
Review Comment:
"find" ?
##########
parquet/src/arrow/schema/extension.rs:
##########
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Arrow Extension Type Support for Parquet
+//!
+//! This module contains mapping code to map Parquet [`LogicalType`]s to/from
+//! Arrow [`ExtensionType`]s.
+//!
+//! Extension types are represented using the metadata from Arrow [`Field`]s
+//! with the key "ARROW:extension:name".
+
+use crate::basic::LogicalType;
+use crate::schema::types::Type;
+use arrow_schema::extension::ExtensionType;
+use arrow_schema::Field;
+
+/// Adds extension type metadata, if necessary, based on the Parquet field's
+/// [`LogicalType`]
+///
+/// Some Parquet logical types, such as Variant, do not map directly to an
+/// Arrow DataType, and instead are represented by an Arrow ExtensionType.
+/// Extension types are attached to Arrow Fields via metadata.
+pub(crate) fn add_extension_type(arrow_field: Field, parquet_type: &Type) -> Field {
+ let result = match parquet_type.get_basic_info().logical_type() {
+ #[cfg(feature = "variant_experimental")]
+ Some(LogicalType::Variant) => {
+ arrow_field.with_extension_type(parquet_variant_compute::VariantType)
+ }
+ // TODO add other LogicalTypes here
+ _ => arrow_field,
+ };
+ result
+}
+
+/// Return the Parquet logical type to use for the specified Arrow field, if any.
+#[cfg(feature = "variant_experimental")]
+pub(crate) fn logical_type_for_struct(field: &Field) -> Option<LogicalType> {
+ use parquet_variant_compute::VariantType;
+ if field.extension_type_name()? == VariantType::NAME {
+ return Some(LogicalType::Variant);
+ };
+ None
Review Comment:
It seems a bit odd that this API call's existence would be gated by the variant feature?
```suggestion
pub(crate) fn logical_type_for_struct(field: &Field) -> Option<LogicalType> {
let type_name = field.extension_type_name()?;
#[cfg(feature = "variant_experimental")]
{
use parquet_variant_compute::VariantType;
if type_name == VariantType::NAME {
return Some(LogicalType::Variant);
}
}
None
```
##########
parquet/src/variant.rs:
##########
@@ -108,6 +98,202 @@
//! ```
//!
//! # Example: Reading a Parquet file with Variant column
-//! (TODO: add example)
+//!
+//! Use the [`VariantType`] extension type to find the Variant column:
+//!
+//! ```
+//! # use std::sync::Arc;
+//! # use std::path::PathBuf;
+//! # use arrow_array::{ArrayRef, RecordBatch, RecordBatchReader};
+//! # use parquet::variant::{Variant, VariantArray, VariantType};
+//! # use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+//! # fn main() -> Result<(), parquet::errors::ParquetError> {
+//! # use arrow_array::StructArray;
+//! # fn file_path() -> PathBuf { // return a testing file path
+//! # PathBuf::from(arrow::util::test_util::parquet_test_data())
+//! # .join("..")
+//! # .join("shredded_variant")
+//! # .join("case-075.parquet")
+//! # }
+//! // Read the Parquet file using standard Arrow Parquet reader
+//! let file = std::fs::File::open(file_path())?;
+//! let mut reader = ArrowReaderBuilder::try_new(file)?.build()?;
+//!
+//! // You can find the Variant using the VariantType extension type
+//! let schema = reader.schema();
+//! let field = schema.field_with_name("var")?;
+//! assert!(field.try_extension_type::<VariantType>().is_ok());
+//!
+//! // The reader will yield RecordBatches with a StructArray
+//! // to convert them to VariantArray, use VariantArray::try_new
+//! let batch = reader.next().unwrap().unwrap();
+//! let col = batch.column_by_name("var").unwrap();
+//! let var_array = VariantArray::try_new(&col)?;
+//! assert_eq!(var_array.len(), 1);
+//! let var_value: Variant = var_array.value(0);
+//! assert_eq!(var_value, Variant::from("iceberg")); // the value in case-075.parquet
+//! # Ok(())
+//! # }
+//! ```
pub use parquet_variant::*;
-pub use parquet_variant_compute as compute;
+pub use parquet_variant_compute::*;
+
+#[cfg(test)]
+mod tests {
+ use crate::arrow::arrow_reader::ArrowReaderBuilder;
+ use crate::arrow::ArrowWriter;
+ use crate::file::metadata::{ParquetMetaData, ParquetMetaDataReader};
+ use crate::file::reader::ChunkReader;
+ use arrow::util::test_util::parquet_test_data;
+ use arrow_array::{ArrayRef, RecordBatch};
+ use arrow_schema::Schema;
+ use bytes::Bytes;
+ use parquet_variant::{Variant, VariantBuilderExt};
+ use parquet_variant_compute::{VariantArray, VariantArrayBuilder, VariantType};
+ use std::path::PathBuf;
+ use std::sync::Arc;
+
+ #[test]
+ fn roundtrip_basic() {
+ roundtrip(variant_array());
+ }
+
+ /// Ensure a file with Variant LogicalType, written by another writer in
+ /// parquet-testing, can be read as a VariantArray
+ #[test]
+ fn read_logical_type() {
Review Comment:
Isn't this the ~same as the doctest above? Do we really need both?
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -154,11 +155,11 @@ fn shredded_get_path(
// Peel away the prefix of path elements that traverses the shredded parts of this variant
// column. Shredding will traverse the rest of the path on a per-row basis.
- let mut shredding_state = input.shredding_state();
+ let mut shredding_state = input.shredding_state().clone();
Review Comment:
Cloning is a cheap reference count bump on the two internal arrays, right?
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -87,20 +88,20 @@ pub(crate) fn follow_shredded_path_element<'a>(
return Ok(missing_path_step());
};
- let field = field
- .as_any()
- .downcast_ref::<ShreddedVariantFieldArray>()
- .ok_or_else(|| {
- // TODO: Should we blow up? Or just end the traversal and let the normal
- // variant pathing code sort out the mess that it must anyway be
- // prepared to handle?
- ArrowError::InvalidArgumentError(format!(
- "Expected a ShreddedVariantFieldArray, got {:?} instead",
- field.data_type(),
- ))
- })?;
-
- Ok(ShreddedPathStep::Success(field.shredding_state()))
+ let struct_array = field.as_struct_opt().ok_or_else(|| {
+ // TODO: Should we blow up? Or just end the traversal and let the normal
+ // variant pathing code sort out the mess that it must anyway be
+ // prepared to handle?
+ ArrowError::InvalidArgumentError(format!(
+ "Expected Struct array while following path, got {}",
+ field.data_type(),
+ ))
+ })?;
+
+ let shredding_state =
+ ShreddedVariantFieldArray::shredding_state_from_struct_array(struct_array)?;
Review Comment:
Why not just `ShreddingState::from`, out of curiosity?
The operation is actually infallible, it just has a fallible signature at the moment.
And AFAIK there's no difference in how we create the shredding state for top-level vs. object field.
##########
parquet/src/arrow/schema/extension.rs:
##########
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Arrow Extension Type Support for Parquet
+//!
+//! This module contains mapping code to map Parquet [`LogicalType`]s to/from
+//! Arrow [`ExtensionType`]s.
+//!
+//! Extension types are represented using the metadata from Arrow [`Field`]s
+//! with the key "ARROW:extension:name".
+
+use crate::basic::LogicalType;
+use crate::schema::types::Type;
+use arrow_schema::extension::ExtensionType;
+use arrow_schema::Field;
+
+/// Adds extension type metadata, if necessary, based on the Parquet field's
+/// [`LogicalType`]
+///
+/// Some Parquet logical types, such as Variant, do not map directly to an
+/// Arrow DataType, and instead are represented by an Arrow ExtensionType.
+/// Extension types are attached to Arrow Fields via metadata.
+pub(crate) fn add_extension_type(arrow_field: Field, parquet_type: &Type) -> Field {
+ let result = match parquet_type.get_basic_info().logical_type() {
+ #[cfg(feature = "variant_experimental")]
+ Some(LogicalType::Variant) => {
+ arrow_field.with_extension_type(parquet_variant_compute::VariantType)
+ }
+ // TODO add other LogicalTypes here
+ _ => arrow_field,
+ };
+ result
+}
+
+/// Return the Parquet logical type to use for the specified Arrow field, if any.
+#[cfg(feature = "variant_experimental")]
+pub(crate) fn logical_type_for_struct(field: &Field) -> Option<LogicalType> {
+ use parquet_variant_compute::VariantType;
+ if field.extension_type_name()? == VariantType::NAME {
+ return Some(LogicalType::Variant);
+ };
+ None
Review Comment:
Ah, there's a cfg-not version below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2362523567
##########
parquet/src/arrow/schema/extension.rs:
##########
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Arrow Extension Type Support for Parquet
+//!
+//! This module contains mapping code to map Parquet [`LogicalType`]s to/from
+//! Arrow [`ExtensionType`]s.
+//!
+//! Extension types are represented using the metadata from Arrow [`Field`]s
+//! with the key "ARROW:extension:name".
+
+use crate::basic::LogicalType;
+use crate::schema::types::Type;
+use arrow_schema::extension::ExtensionType;
+use arrow_schema::Field;
+
+/// Adds extension type metadata, if necessary, based on the Parquet field's
+/// [`LogicalType`]
+///
+/// Some Parquet logical types, such as Variant, do not map directly to an
+/// Arrow DataType, and instead are represented by an Arrow ExtensionType.
+/// Extension types are attached to Arrow Fields via metadata.
+pub(crate) fn add_extension_type(arrow_field: Field, parquet_type: &Type) -> Field {
+ let result = match parquet_type.get_basic_info().logical_type() {
+ #[cfg(feature = "variant_experimental")]
+ Some(LogicalType::Variant) => {
+ arrow_field.with_extension_type(parquet_variant_compute::VariantType)
+ }
+ // TODO add other LogicalTypes here
Review Comment:
They are interleaved in the schema conversion code with `cfg` like this:
https://github.com/apache/arrow-rs/blob/c2a1fcabdf283b071742415fe262e35d1ded4c67/parquet/src/arrow/schema/mod.rs#L405-L412
I plan to move the code handling the other types into the extension.rs module as a follow on PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2362704747
##########
parquet/src/arrow/schema/extension.rs:
##########
@@ -0,0 +1,62 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Arrow Extension Type Support for Parquet
+//!
+//! This module contains mapping code to map Parquet [`LogicalType`]s to/from
+//! Arrow [`ExtensionType`]s.
+//!
+//! Extension types are represented using the metadata from Arrow [`Field`]s
+//! with the key "ARROW:extension:name".
+
+use crate::basic::LogicalType;
+use crate::schema::types::Type;
+use arrow_schema::extension::ExtensionType;
+use arrow_schema::Field;
+
+/// Adds extension type metadata, if necessary, based on the Parquet field's
+/// [`LogicalType`]
+///
+/// Some Parquet logical types, such as Variant, do not map directly to an
+/// Arrow DataType, and instead are represented by an Arrow ExtensionType.
+/// Extension types are attached to Arrow Fields via metadata.
+pub(crate) fn add_extension_type(arrow_field: Field, parquet_type: &Type) -> Field {
+ let result = match parquet_type.get_basic_info().logical_type() {
+ #[cfg(feature = "variant_experimental")]
+ Some(LogicalType::Variant) => {
+ arrow_field.with_extension_type(parquet_variant_compute::VariantType)
+ }
+ // TODO add other LogicalTypes here
+ _ => arrow_field,
+ };
+ result
+}
+
+/// Return the Parquet logical type to use for the specified Arrow field, if any.
+#[cfg(feature = "variant_experimental")]
+pub(crate) fn logical_type_for_struct(field: &Field) -> Option<LogicalType> {
+ use parquet_variant_compute::VariantType;
+ if field.extension_type_name()? == VariantType::NAME {
+ return Some(LogicalType::Variant);
+ };
+ None
Review Comment:
That sounds like a flaw in the API. Should it return `Option<Result<E>>` to cover the "not an extension type at all" case? Either way, people have to manually check _something_.
BTW, the JSON extension type handling for string columns is doing the unconditional `try_extension_type`, so maybe we should fix that one as well? (either by the API change or by adding a specific check for existence of metadata before making the `try_extension_type` call)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2355935781
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -246,6 +288,57 @@ impl VariantArray {
pub fn typed_value_field(&self) -> Option<&ArrayRef> {
self.shredding_state.typed_value_field()
}
+
+ // TODO return a field with the correct metadata?
+ pub fn data_type(&self) -> &DataType {
+ self.inner.data_type()
+ }
+
+ pub fn slice(&self, offset: usize, length: usize) -> Self {
+ let inner = self.inner.slice(offset, length);
+ let metadata = self.metadata.slice(offset, length);
+ let shredding_state = self.shredding_state.slice(offset, length);
+ Self {
+ inner,
+ metadata,
+ shredding_state,
+ }
+ }
+
+ pub fn len(&self) -> usize {
+ self.inner.len()
+ }
+
+ pub fn is_empty(&self) -> bool {
+ self.inner.is_empty()
+ }
+
+ pub fn nulls(&self) -> Option<&NullBuffer> {
+ self.inner.nulls()
+ }
+
+ /// Is the element at index null?
+ pub fn is_null(&self, index: usize) -> bool {
+ // TODO: should we account for shredded nulls here?
Review Comment:
> Do you mean, return true here if `Variant::Null` in the `value` column and `NULL` in the `typed_value`?
Yeah, that is what I was thinking
> I think that's more a job for `logical_nulls`?
that is a good call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2355934091
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -617,59 +710,6 @@ fn typed_value_to_variant(typed_value: &ArrayRef, index: usize) -> Variant<'_, '
}
}
-impl Array for VariantArray {
Review Comment:
Yes, we do, but when I tried that was a more significant change, so I will probably leave it for a follow on
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2356687243
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -24,12 +24,54 @@ use arrow::datatypes::{
Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
UInt32Type, UInt64Type, UInt8Type,
};
+use arrow_schema::extension::ExtensionType;
use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Uuid;
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
+/// Variant Canonical Extension Type
+pub struct VariantType;
+
+impl ExtensionType for VariantType {
+ const NAME: &'static str = "parquet.variant";
+
+ // Variants have no extension metadata
+ type Metadata = ();
+
+ fn metadata(&self) -> &Self::Metadata {
+ &()
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ None
+ }
+
+ fn deserialize_metadata(_metadata: Option<&str>) -> Result<Self::Metadata, ArrowError> {
+ Ok(())
+ }
+
+ fn supports_data_type(&self, data_type: &DataType) -> Result<(), ArrowError> {
+ // Note don't check for metadata/value fields here because they may be
+ // absent in shredded variants
+ if matches!(data_type, DataType::Struct(_)) {
+ Ok(())
Review Comment:
There are a few cases:
* Pass `as_type=None` -- this would preserve existing shredding structure, if any
* Pass `as_type=Some(... VariantType...)` -- what schema should the underlying struct have, to:
* Force shredding as a specific schema
* Force unshredding to binary
* Return whatever was there (equivalent to passing None)
* Pass `as_type=Some(StructType(... VariantType ...))` -- the top-level variant is shredded as struct, but the field(s) inside that struct are variant-typed (not shredded). The same three questions apply, but now `as_type=None` is not a fallback
* We had talked previously about a providing function that could return the shredding schema for a given path, so the user can stub in the variant-typed fields however they want. But that's pretty fiddly.
* The alternative (which I proposed above) was to allow annotating the variant type to indicate that `variant_get` should preserve the underlying shredding structure, without having to actually specify it.
* Much cleaner, and actually simpler for the implementation as well. Once the field's path is exhausted, just return whatever was there (similar to how `as_type=None` is implemented today); if we passed an explicit shredding schema instead, then the implementation would have to examine every node of the schema, all the way to leaf level, in case the provided shredding structure doesn't actually match the underlying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#issuecomment-3309685127
> > > the parquet reader will read Binary columns as Binary rather than BinaryView, which is somewhat awkward.
> >
> >
> > Should we consider just adding support for Binary columns?
>
> I think we will need to do so eventually, given the arrow spec explicitly says they are supported
>
> However, I think we can add that support in parallel (I would like to avoid blocking logical type support on Binary support)
Tracking with https://github.com/apache/arrow-rs/issues/8387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#issuecomment-3303591242
> > the parquet reader will read Binary columns as Binary rather than BinaryView, which is somewhat awkward.
>
> Should we consider just adding support for Binary columns?
I think we will need to do so eventually, given the arrow spec explicitly says they are supported
However, I think we can add that support in parallel (I would like to avoid blocking logical type support on Binary support)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2363607508
##########
parquet/src/arrow/arrow_reader/mod.rs:
##########
@@ -2295,7 +2295,7 @@ mod tests {
let batch = record_reader.next().unwrap().unwrap();
assert_eq!(batch.num_rows(), 1);
- let expected_schema = Schema::new(Fields::from(vec![Field::new(
+ let expected_schema = Schema::new(vec![Field::new(
Review Comment:
Made its own PR
- https://github.com/apache/arrow-rs/pull/8391
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2363716311
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -154,11 +155,11 @@ fn shredded_get_path(
// Peel away the prefix of path elements that traverses the shredded parts of this variant
// column. Shredding will traverse the rest of the path on a per-row basis.
- let mut shredding_state = input.shredding_state();
+ let mut shredding_state = input.shredding_state().clone();
Review Comment:
Yes. Let's move this conversation to
- https://github.com/apache/arrow-rs/pull/8392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Path Finding: Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2364659371
##########
parquet/src/variant.rs:
##########
@@ -108,6 +98,202 @@
//! ```
//!
//! # Example: Reading a Parquet file with Variant column
-//! (TODO: add example)
+//!
+//! Use the [`VariantType`] extension type to find the Variant column:
+//!
+//! ```
+//! # use std::sync::Arc;
+//! # use std::path::PathBuf;
+//! # use arrow_array::{ArrayRef, RecordBatch, RecordBatchReader};
+//! # use parquet::variant::{Variant, VariantArray, VariantType};
+//! # use parquet::arrow::arrow_reader::ArrowReaderBuilder;
+//! # fn main() -> Result<(), parquet::errors::ParquetError> {
+//! # use arrow_array::StructArray;
+//! # fn file_path() -> PathBuf { // return a testing file path
+//! # PathBuf::from(arrow::util::test_util::parquet_test_data())
+//! # .join("..")
+//! # .join("shredded_variant")
+//! # .join("case-075.parquet")
+//! # }
+//! // Read the Parquet file using standard Arrow Parquet reader
+//! let file = std::fs::File::open(file_path())?;
+//! let mut reader = ArrowReaderBuilder::try_new(file)?.build()?;
+//!
+//! // You can find the Variant using the VariantType extension type
+//! let schema = reader.schema();
+//! let field = schema.field_with_name("var")?;
+//! assert!(field.try_extension_type::<VariantType>().is_ok());
+//!
+//! // The reader will yield RecordBatches with a StructArray
+//! // to convert them to VariantArray, use VariantArray::try_new
+//! let batch = reader.next().unwrap().unwrap();
+//! let col = batch.column_by_name("var").unwrap();
+//! let var_array = VariantArray::try_new(&col)?;
+//! assert_eq!(var_array.len(), 1);
+//! let var_value: Variant = var_array.value(0);
+//! assert_eq!(var_value, Variant::from("iceberg")); // the value in case-075.parquet
+//! # Ok(())
+//! # }
+//! ```
pub use parquet_variant::*;
-pub use parquet_variant_compute as compute;
+pub use parquet_variant_compute::*;
+
+#[cfg(test)]
+mod tests {
+ use crate::arrow::arrow_reader::ArrowReaderBuilder;
+ use crate::arrow::ArrowWriter;
+ use crate::file::metadata::{ParquetMetaData, ParquetMetaDataReader};
+ use crate::file::reader::ChunkReader;
+ use arrow::util::test_util::parquet_test_data;
+ use arrow_array::{ArrayRef, RecordBatch};
+ use arrow_schema::Schema;
+ use bytes::Bytes;
+ use parquet_variant::{Variant, VariantBuilderExt};
+ use parquet_variant_compute::{VariantArray, VariantArrayBuilder, VariantType};
+ use std::path::PathBuf;
+ use std::sync::Arc;
+
+ #[test]
+ fn roundtrip_basic() {
+ roundtrip(variant_array());
+ }
+
+ /// Ensure a file with Variant LogicalType, written by another writer in
+ /// parquet-testing, can be read as a VariantArray
+ #[test]
+ fn read_logical_type() {
Review Comment:
Or, just move the assertions to the doc test, hidden behind `#` if users don't care to see them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2362516196
##########
parquet-variant-compute/src/variant_get.rs:
##########
@@ -1651,7 +1633,7 @@ mod test {
}
}
- Arc::new(builder.build())
+ builder.build().into()
Review Comment:
In my opinion it was easier to understand as the type is explicitly listed.
However, I think it is just an opinion and I know others prefer the `into` method.
I don't think there was any reason for the discrepancy in this PR -- I'll try and make them the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Path Finding: Support reading/writing `VariantArray` to parquet with `Variant` LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#issuecomment-3313196572
I broke out the first chunk of work into its own PR here:
- https://github.com/apache/arrow-rs/pull/8392
Once that is merged I'll make a PR with the remaining work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support reading/writing `VariantArray` to parquet with correct LogicalType [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8365:
URL: https://github.com/apache/arrow-rs/pull/8365#discussion_r2360625046
##########
parquet-variant-compute/src/variant_array.rs:
##########
@@ -24,12 +24,54 @@ use arrow::datatypes::{
Float16Type, Float32Type, Float64Type, Int16Type, Int32Type, Int64Type, Int8Type, UInt16Type,
UInt32Type, UInt64Type, UInt8Type,
};
+use arrow_schema::extension::ExtensionType;
use arrow_schema::{ArrowError, DataType, Field, FieldRef, Fields};
use parquet_variant::Uuid;
use parquet_variant::Variant;
use std::any::Any;
use std::sync::Arc;
+/// Variant Canonical Extension Type
+pub struct VariantType;
+
+impl ExtensionType for VariantType {
+ const NAME: &'static str = "parquet.variant";
+
+ // Variants have no extension metadata
+ type Metadata = ();
+
+ fn metadata(&self) -> &Self::Metadata {
+ &()
+ }
+
+ fn serialize_metadata(&self) -> Option<String> {
+ None
+ }
+
+ fn deserialize_metadata(_metadata: Option<&str>) -> Result<Self::Metadata, ArrowError> {
+ Ok(())
+ }
+
+ fn supports_data_type(&self, data_type: &DataType) -> Result<(), ArrowError> {
+ // Note don't check for metadata/value fields here because they may be
+ // absent in shredded variants
+ if matches!(data_type, DataType::Struct(_)) {
+ Ok(())
Review Comment:
> Once the field's path is exhausted, just return whatever was there (similar to how as_type=None is implemented today); if we passed an explicit shredding schema instead, then the implementation would have to examine every node of the schema, all the way to leaf level, in case the provided shredding structure doesn't actually match the underlying.
This is sort of what I expected to happen today.
Maybe we could use the `cast_to_variant` kernel for this? For example, if I had a VariantArray with some arbitrary shredding, I could do:
```rust
let array: VariantArray = ...; // input somehow
// get the "foo" field (may be shredded).
let foo = variant_get(array, Path::from("foo"))?
// ensure foo is non shredded
let unshredded_foo = cast_to_variant(foo)?;
```
🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org