You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "sdf-jkl (via GitHub)" <gi...@apache.org> on 2026/07/23 19:40:56 UTC
[I] [Docs][Format] Variant extension: align typed_value primitive type mappings with the Parquet shredding spec [arrow]
sdf-jkl opened a new issue, #50622:
URL: https://github.com/apache/arrow/issues/50622
### Describe the enhancement requested
The Primitive Type Mappings table in the Variant canonical extension is the normative
list of Arrow types allowed for `typed_value`. The only place the table is referenced
from is the `typed_value` storage rule, so its whole role is defining legal `typed_value`
types:
https://github.com/apache/arrow/blob/f10c93c000775c5fb30ca230f540358611091589/docs/source/format/CanonicalExtensions.rst?plain=1#L453
The table appears to be derived from the Variant *encoding* types table
(https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types),
but its role corresponds to the shredding spec's Shredded Value Types table
(https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types),
which defines what a `typed_value` column may be. The two tables differ, so the Arrow
table currently allows storage with no valid Parquet shredded representation:
- `Uint8 -> Int16`, `Uint16 -> Int32`, `Uint32 -> Int64`: the shredding spec only permits
signed integer `typed_value` columns, so unsigned Arrow storage can never round trip
(`Arrow Uint -> Parquet INT -> Arrow Int`). A `Uint8` column also cannot represent the
full value domain of its claimed variant type (int16), so it can only act as a partial
storage that forces fallback to `value`. See apache/arrow-rs#10416 for discussion.
- `Null -> Null`: the shredding table has no null row; a variant null must be encoded in
the `value` column, so a `Null` typed_value column has no Parquet representation either.
In practice the unsigned path already produces non-conformant files: Arrow Go writes
unsigned `typed_value` columns to Parquet with `INT(8/16/32, false)` annotations, which
the shredding spec does not allow.
Proposal: re-derive the table from the shredding spec's Shredded Value Types table. Each
row maps one variant type to its Parquet shredded type and the Arrow storage type(s) able
to represent its full value domain. The table then defines reconstruction (a `typed_value`
column of the listed Arrow type holds values of exactly the listed variant type) and the
write-side lowering (the Parquet column is the only valid Parquet representation).
| Variant type | Parquet type | Arrow `typed_value` type |
|-----------------|-------------------------------------------------|-----------------------------------|
| boolean | BOOLEAN | Boolean |
| int8 | INT32, INT(8, true) | Int8 |
| int16 | INT32, INT(16, true) | Int16 |
| int32 | INT32 | Int32 |
| int64 | INT64 | Int64 |
| float | FLOAT | Float32 |
| double | DOUBLE | Float64 |
| decimal4 | INT32, DECIMAL(P, S) | Decimal32(P, S), 1 <= P <= 9 |
| decimal8 | INT64, DECIMAL(P, S) | Decimal64(P, S), 10 <= P <= 18 |
| decimal16 | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY, DECIMAL(P, S)| Decimal128(P, S), 19 <= P <= 38 |
| date | INT32, DATE | Date32 |
| time | INT64, TIME(false, MICROS) | Time64(us) |
| timestamptz(6) | INT64, TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
| timestamptz(9) | INT64, TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
| timestampntz(6) | INT64, TIMESTAMP(false, MICROS) | Timestamp(us) |
| timestampntz(9) | INT64, TIMESTAMP(false, NANOS) | Timestamp(ns) |
| binary | BYTE_ARRAY | Binary / LargeBinary / BinaryView |
| string | BYTE_ARRAY, STRING | Utf8 / LargeUtf8 / Utf8View |
| uuid | FIXED_LEN_BYTE_ARRAY[len=16], UUID | UUID extension type |
The decimal precision bands follow the decimal table in
[VariantEncoding.md](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types):
precision alone selects the physical type (the bands are disjoint, so the narrowest type
is required), and scale must satisfy `0 <= S <= P`.
This also resolves three existing ambiguities: the current `Time64` row gains its required
unit (variant time is micros only), the timestamp rows gain explicit units, and the
decimal rows gain their precision bounds.
Related: #50620, apache/parquet-format#591
### Component(s)
Format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5072154398
> For arrow-cpp, the Parquet writer serializes the Arrow schema into footer when the writer property store_schema is enabled (which is the default value) and then Parquet reader deserializes that schema to restore their original form on read.
arrow-go does the same using the same property information and is thus compatible with arrow-cpp (and the impls based on it like pyarrow) writing the arrow schema into the footer.
> In practice the unsigned path already produces non-conformant files: Arrow Go writes unsigned typed_value columns to Parquet with INT(8/16/32, false) annotations, which the shredding spec does not allow.
Can you file this as an issue on arrow-go so we can track and fix this? We should definitely make sure that arrow-go is spec conformant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5072874853
> > For arrow-cpp, the Parquet writer serializes the Arrow schema into footer when the writer property store_schema is enabled (which is the default value) and then Parquet reader deserializes that schema to restore their original form on read.
>
> arrow-go does the same using the same property information and is thus compatible with arrow-cpp (and the impls based on it like pyarrow) writing the arrow schema into the footer.
Arrow-rs does the same thing
It looks to me like @sdf-jkl observation is that there are certain arrow types that can not be round tripped through a Parquet Variant column without some sort of additional metadata
Specifically something like `UInt8` --> `Variant` must be stored as Variant::Int16 (so the knowledge that the column only has UInt8 values is lost)
As mentioned above, the normal Parquet logical types have the same problem, so the C++ writer adds extra metadata in the file so even though a value might be stored as Parquet type Int 16, Arrow readers know to try and read it as UInt8. This metadata convention seems to be commonly followed.
Maybe we can think about additional metadata writers could add if they want to recreate the original schema.
I am not sure how important this case is, though, because if you had data that was originally highly typed and structured you probably wouldn't be using Variant 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5094693400
@HuaHuaY yes, that makes sense. I updated the proposed table in the issue description, the precision bounds now constrain both the Parquet and the Arrow columns.
There is a related gap upstream: the shredding spec never states the decimal precision rules at all. The bands only appear in the [Decimal table in VariantEncoding.md](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types), while the decimal rows in VariantShredding.md are unbounded DECIMAL(P, S). So the Arrow table has to carry the bounds explicitly, and VariantShredding.md could use the same clarification or a cross-reference.
> So, is the final conclusion that we should update the Arrow documentation? How should we go about driving this forward?
Update the Arrow doc and make an upstream Parquet PR to add decimal precision clarification/cross-reference to the shredding schema.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5074576968
> I am proposing to drop unsigned integers from the typed_value allowed types for Arrow since they always have to be written as signed.
so is the proposed fix just an additional caveat to the spec that says typed_value has to be an arrow type that has a 1:1 mapping with the parquet types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5184451380
I filed a PR with the proposed changes - https://github.com/apache/arrow/pull/50810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5089341753
So, is the final conclusion that we should update the Arrow documentation? How should we go about driving this forward?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5090606277
I’ve found another confusing issue. The Arrow specification maps `decimal32` to `decimal4`, `decimal64` to `decimal8`, `decimal128` to `decimal16`. However, the Parquet documentation maps types based on decimal precision. Given the earlier discussion, I feel we should defer to the Parquet standard here as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5122022807
@alamb @zeroshade @wgtmac Please confirm if you think this is the correct path and I'll start working on the Arrow spec PR and submit an issue in arrow-go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5065701101
> unsigned Arrow storage can never round trip (Arrow Uint -> Parquet INT -> Arrow Int)
I think this is possible. I'm not familiar with arrow-rs. For arrow-cpp, the Parquet writer serializes the Arrow schema into footer when the writer property `store_schema` is enabled (which is the default value) and then Parquet reader deserializes that schema to restore their original form on read.
> Null -> Null: the shredding table has no null row
This is a good argument. Parquet indeed has a UNKNOWN logical type to represent a column with no known type. It would be useful when user-hinted column cannot be shredded. So it seems like a missing piece in the Parquet shredding spec?
cc @zeroshade @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5074592899
> > I am proposing to drop unsigned integers from the typed_value allowed types for Arrow since they always have to be written as signed.
>
> so is the proposed fix just an additional caveat to the spec that says typed_value has to be an arrow type that has a 1:1 mapping with the parquet types?
Yes, that would be it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #50622:
URL: https://github.com/apache/arrow/issues/50622#issuecomment-5073509393
> I am not sure how important this case is, though, because if you had data that was originally highly typed and structured you probably wouldn't be using Variant 🤔
@alamb That's the point I was trying to make. Variant's physical types only include signed integers, same goes for the `typed_value` type support in the Shredding spec.
I am proposing to drop unsigned integers from the `typed_value` allowed types for Arrow since they always have to be written as signed.
> > Null -> Null: the shredding table has no null row
>
> This is a good argument. Parquet indeed has a UNKNOWN logical type to represent a column with no known type. It would be useful when user-hinted column cannot be shredded. So it seems like a missing piece in the Parquet shredding spec?
@wgtmac Same goes for the Null type, I'm proposing to drop it from the Arrow table rather than add a null row to the Parquet spec. A null in `typed_value` already carries meaning: the row's value couldn't be shredded into the column type (or the field is missing, for object fields). An UNKNOWN column is always null, so it could never mark a row as shredded, and a Variant null would still have to be encoded in `value` as `00`. Worse, for object fields `typed_value` null + `value` null means the field is missing, so shredding null into `typed_value` would make "field present and null" indistinguishable from "field missing". I don't think the null row is a missing piece of the shredding spec, that state is reserved by the null-signaling protocol itself.
> Can you file this as an issue on arrow-go so we can track and fix this? We should definitely make sure that arrow-go is spec conformant
@zeroshade before filing I want to pin the correct behavior. I think Arrow should not support unsigned int `typed_value` at all, which would make the fix different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #50622: [Docs][Format] Variant extension: align `typed_value` primitive type mappings with the Parquet shredding spec
URL: https://github.com/apache/arrow/issues/50622
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org