You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/11/25 17:58:37 UTC
[I] [Variant] easier way to construct a shredded schema [arrow-rs]
alamb opened a new issue, #8922:
URL: https://github.com/apache/arrow-rs/issues/8922
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Filing this for a friend (@XiangpengHao !)
arrow-rs contains a kernel to "shred" a variant value: [`shred_variant`](https://docs.rs/parquet/latest/parquet/variant/fn.shred_variant.html)
However, to call this method you need to create the entire shredded schema, which is complicated. For example to shred a `time` and host field you would need to provide something like:
```rust
{
metadata: BINARY,
value: BINARY,
typed_value: {
time: {
value: BINARY,
typed_value: Timestamp,
},
hostname: {
value: BINARY,
typed_value: String,
},
}
}
```
**Describe the solution you'd like**
I would like it to be easier to construct shredded variants for common cases
**Describe alternatives you've considered**
@XiangpengHao suggested to me in person that a similar API to [`variant_get`](https://docs.rs/parquet/latest/parquet/variant/fn.variant_get.html) would be natural. You would explicitly specify which path should be shredded as what type
For example, it would be nice to create the schema above by specifying only `time` and `host`. Something like
```rust
let schema = VariantSchemaBuilder::default()
.with_path("time", &DataType::Timestamp(Nanoseconds, None));
.with_path("hostname", &DataType::Utf8)
.build();
// Pass in the shredded schema
let shredded_array = shred_variant(&input, &schema)?;
```
It turns out (naturally) that he implemented just such an API in liquid cache that I think is worth considering upstream:
https://github.com/XiangpengHao/liquid-cache/blob/33dbaaaec3de5207885e778e57d14df2fb69071f/src/storage/src/utils/variant_schema.rs#L14
Here is an example of it working: https://github.com/XiangpengHao/liquid-cache/blob/33dbaaaec3de5207885e778e57d14df2fb69071f/src/storage/src/utils/variant_schema.rs#L162-L178
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3576852727
FYI @klion26 and @scovich
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3599120078
Any reason we can't just translate a normal schema tho? DataType::Struct -> variant object, DataType::List -> variant array, etc?
If we want a general schema builder, that's a separate thing IMO (useful outside variant context)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3585297244
Maybe I can help implement it next week, if no one takes it by then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3716625720
`label_issue.py` automatically added labels {'parquet-variant'} from #8940
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3612892779
> Any reason we can't just translate a normal schema tho? DataType::Struct -> variant object, DataType::List -> variant array, etc?
I don't understand this proposal. If a user already has a Struct (or data that is already is well formed and can be parsed to a Struct via JSON), why would they convert it to a Variant 🤔 ?
Maybe the idea is that users would describe the schema the *want* and then parse some JSON and using this shredded schema would handle exceptions 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3596839229
I can work on this as it should be easy to port to upstream
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3585086992
Currently, `shred_variant(array: &VariantArray, as_type: &DataType)` needs to pass in a `DataType`, it may become difficult for some `DataType::Struct`, the new builder is more convenient for this. +1 for adding this.
PS, the old interface receives the `DataType` as a parameter, it's more arrow-native(`DataType/Field` are types already in arrow-rs), not sure if we need to keep it or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8922: [Variant] easier way to construct a shredded schema
URL: https://github.com/apache/arrow-rs/issues/8922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3614447739
> > Any reason we can't just translate a normal schema tho? DataType::Struct -> variant object, DataType::List -> variant array, etc?
>
> I don't understand this proposal. If a user already has a Struct (or data that is already is well formed and can be parsed to a Struct via JSON), why would they convert it to a Variant 🤔 ?
>
> Maybe the idea is that users would describe the schema the _want_ and then parse some JSON and using this shredded schema would handle exceptions 🤔
That was precisely my understanding? I say "I _think_ (or I _want_) data with the following schema: `<schema>`" and the helper produces the corresponding shredding schema (with `value` and `typed_value` columns etc). And then if the variant data don't actually fit the schema, it just lands in `value` instead of `typed_value`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "XiangpengHao (via GitHub)" <gi...@apache.org>.
XiangpengHao commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3596839739
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] easier way to construct a shredded schema [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on issue #8922:
URL: https://github.com/apache/arrow-rs/issues/8922#issuecomment-3599112776
Agree users should be able to reason in terms of the logical shredding schema they imagine, not the physical shredding schema the spec translates that to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org