You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "HuaHuaY (via GitHub)" <gi...@apache.org> on 2026/07/23 13:47:30 UTC
[PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
HuaHuaY opened a new pull request, #10417:
URL: https://github.com/apache/arrow-rs/pull/10417
# Which issue does this PR close?
Closes #10416.
# Rationale for this change
Arrow allows mapping Uint8/16/32 to Parquet types Int16/32/64. However, the current code does not support this.
Arrow's documentation: https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings
# What changes are included in this PR?
- Support `UInt8`, `UInt16`, and `UInt32` Arrow Variant `typed_value` fields.
- Continue rejecting unsigned types in Parquet Variant schemas.
- **Add lossless `UInt32` Arrow array to Parquet `INT64` writer coercion**
# Are these changes tested?
Yes.
# Are there any user-facing changes?
`VariantArray` and `shred_variant` now support canonical `UInt8`/`UInt16`/`UInt32` `typed_value` fields.
**`ArrowWriter` can also write `UInt32` arrays to `INT64` columns when using a compatible custom Parquet schema.** This is because it's not convenient to distinguish whether the column is in a variant within the context of `arrow_writer/mod.rs`. I think this is a side effect of this PR; please point out any better suggestions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #10417:
URL: https://github.com/apache/arrow-rs/pull/10417#issuecomment-5305334292
@Jefffrey I'm waiting for my arrow docs PR to get merged/reviewed. This PR can wait till then.
The PR - https://github.com/apache/arrow/pull/50810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY closed pull request #10417: variant: Support Arrow UInt8/UInt16/UInt32 typed values
URL: https://github.com/apache/arrow-rs/pull/10417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #10417:
URL: https://github.com/apache/arrow-rs/pull/10417#issuecomment-5062749697
@HuaHuaY I submitted an issue here https://github.com/apache/arrow/issues/50622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on PR #10417:
URL: https://github.com/apache/arrow-rs/pull/10417#issuecomment-5761907835
The standard has been revised. So close this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on PR #10417:
URL: https://github.com/apache/arrow-rs/pull/10417#issuecomment-5059225094
Hi @alamb. When implementing variant code for the Arrow cpp repository, I noticed inconsistencies between arrow-rs and the arrow specification. However, I am not familiar with the developers of the arrow-rs repository. Could you please review the code or tag the relevant developers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on PR #10417:
URL: https://github.com/apache/arrow-rs/pull/10417#issuecomment-5060617874
> Hi @HuaHuaY!
>
> Arrow Variant closely follows the Parquet Variant spec - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
>
> The arrow [primitive type mapping](https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings) you've linked is following the table from the spec here - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types - It's about unshredded Variant.
>
> The shredding spec here - https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types - lists which Parquet types should be mapped to each Variant type for shredding.
>
> Variant Type Parquet Physical Type Parquet Logical Type
> ...
> int8 INT32 INT(8, true)
> int16 INT32 INT(16, true)
> int32 INT32
> int64 INT64
> ...
> Here Int* Variant types have to be represented by signed integers in Parquet -> we use signed int arrow arrays to store them.
>
> cc @klion26
Hi @sdf-jkl!
As I understand it, the documentation you provided regarding "unshredded" and "shredded" variants refers to the mapping between primitive types and Parquet types within the Parquet variant format. It does not address how Arrow's variant type maps to the Parquet variant. Notably, the Arrow documentation mentions unsigned integers, whereas the Parquet documentation does not. This PR specifically concerns unsigned integer types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #10417:
URL: https://github.com/apache/arrow-rs/pull/10417#issuecomment-5060497142
Hi @HuaHuaY!
Arrow Variant closely follows the Parquet Variant spec - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
The arrow [primitive type mapping](https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings) you've linked is following the table from the spec here - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types - It's about unshredded Variant.
The shredding spec here - https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types - lists which Parquet types should be mapped to each Variant type for shredding.
| Variant Type | Parquet Physical Type | Parquet Logical Type |
|-----------------------------|-----------------------------------|--------------------------|
...
| int8 | INT32 | INT(8, true) |
| int16 | INT32 | INT(16, true) |
| int32 | INT32 | |
| int64 | INT64 | |
...
Here Int* Variant types have to be represented by signed integers in Parquet -> we use signed int arrow arrays to store them.
cc @klion26
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10417:
URL: https://github.com/apache/arrow-rs/pull/10417#issuecomment-5305117882
so whats the status of this PR, is this something that needs to wait on upstream discussion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] variant: Support Arrow UInt8/UInt16/UInt32 typed values [arrow-rs]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #10417:
URL: https://github.com/apache/arrow-rs/pull/10417#issuecomment-5061148234
@HuaHuaY you are correct! I was reading more into it and saw where I'm wrong.
> The arrow [primitive type mapping](https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings) you've linked is following the table from the spec here - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types - It's about unshredded Variant.
This point I made was incorrect and the arrow spec indeed says: "A field named `typed_value` which can be a [Primitive Type Mappings](https://arrow.apache.org/docs/format/CanonicalExtensions.html#variant-primitive-type-mapping) or a `List`, `LargeList`, `ListView` or `Struct`"
So the PR and the issue make sense given the spec, **but** I wonder if the spec itself should support `Uint` for typed value.
`Arrow Uint -> Parquet INT -> Arrow Int`
Given any UInt will be written as a signed INT Parquet type and never read back as UInt, it never round trips. The only places a UInt typed_value can come from are in-memory construction or IPC.
I'll raise a question on the Arrow mailing list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org