You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "sdf-jkl (via GitHub)" <gi...@apache.org> on 2026/08/04 20:46:19 UTC
[PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
sdf-jkl opened a new pull request, #50810:
URL: https://github.com/apache/arrow/pull/50810
### Rationale for this change
The existing Arrow spec is misleading and allows certain Arrow types to be typed value Variant fields. The Parquet Shredding spec specifically rejects them - Unsigned integers, Null.
Decimal mapping needs clarification about precision.
### What changes are included in this PR?
Change the Primitive Type Mappings table for Parquet Variant extension type.
Rendered version - https://github.com/sdf-jkl/arrow/blob/gh50622-variant-typed-value-mappings/docs/source/format/CanonicalExtensions.rst#primitive-type-mappings
### Are these changes tested?
N/A
### Are there any user-facing changes?
Changes to the Arrow docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3798158456
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,82 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It is derived from the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification: each row maps a Variant
+primitive type to the Parquet type required for a shredded ``typed_value``
+column (physical type, followed by the logical type annotation if any) and to
+the Arrow type(s) able to represent that Variant type's full value domain.
+A ``typed_value`` field of one of the listed Arrow types holds values of
+exactly the corresponding Variant type, and the listed Parquet type is its
+only valid Parquet representation.
+
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| Variant Type | Parquet Type | Arrow ``typed_value`` Type |
++========================================+==================================================+=============================================+
+| boolean | BOOLEAN | Boolean |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int8 | INT32, INT(8, true) | Int8 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int16 | INT32, INT(16, true) | Int16 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int32 | INT32 | Int32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int64 | INT64 | Int64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| float | FLOAT | Float32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| double | DOUBLE | Float64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32, DECIMAL(P, S) | Decimal32(P, S) |
Review Comment:
https://parquet.apache.org/docs/file-format/types/variantencoding/ says
> The Decimal type contains a scale, but no precision. The implied precision of a decimal value is floor(log_10(val)) + 1.
I don't unersatnd how that maps to `1 <= P <= 9, 0 <= S <= P` 🤔
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,82 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It is derived from the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification: each row maps a Variant
+primitive type to the Parquet type required for a shredded ``typed_value``
+column (physical type, followed by the logical type annotation if any) and to
+the Arrow type(s) able to represent that Variant type's full value domain.
+A ``typed_value`` field of one of the listed Arrow types holds values of
+exactly the corresponding Variant type, and the listed Parquet type is its
+only valid Parquet representation.
+
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| Variant Type | Parquet Type | Arrow ``typed_value`` Type |
Review Comment:
Perhaps we could also add a reference to the location of the parquet and arrow type lists
I think the arrow ones are here:
https://arrow.apache.org/docs/format/Columnar.html#data-types
The parquet ones are here:
https://parquet.apache.org/docs/file-format/types/
https://parquet.apache.org/docs/file-format/types/logicaltypes/
I am a little confused about what (logical or physical) type is used in the Parquet Type column. Some seem to be just the physical type (`INT64`) , and some seem to have both logical and physical (`INT32, INT(8, true) `). It would help to be spcific about what the Parquet type column represents and use it consistently
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,82 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It is derived from the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification: each row maps a Variant
+primitive type to the Parquet type required for a shredded ``typed_value``
+column (physical type, followed by the logical type annotation if any) and to
+the Arrow type(s) able to represent that Variant type's full value domain.
+A ``typed_value`` field of one of the listed Arrow types holds values of
+exactly the corresponding Variant type, and the listed Parquet type is its
+only valid Parquet representation.
+
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| Variant Type | Parquet Type | Arrow ``typed_value`` Type |
++========================================+==================================================+=============================================+
+| boolean | BOOLEAN | Boolean |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int8 | INT32, INT(8, true) | Int8 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int16 | INT32, INT(16, true) | Int16 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int32 | INT32 | Int32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int64 | INT64 | Int64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| float | FLOAT | Float32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| double | DOUBLE | Float64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32, DECIMAL(P, S) | Decimal32(P, S) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal8 (10 <= P <= 18, 0 <= S <= P) | INT64, DECIMAL(P, S) | Decimal64(P, S) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal16 (19 <= P <= 38, 0 <= S <= P) | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY, DECIMAL(P, S) | Decimal128(P, S) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| date | INT32, DATE | Date32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| time | INT64, TIME(false, MICROS) | Time64(us) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| timestamptz(6) | INT64, TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| timestamptz(9) | INT64, TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| timestampntz(6) | INT64, TIMESTAMP(false, MICROS) | Timestamp(us) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| timestampntz(9) | INT64, TIMESTAMP(false, NANOS) | Timestamp(ns) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| binary | BYTE_ARRAY | Binary / LargeBinary / BinaryView |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| string | BYTE_ARRAY, STRING | String / LargeString / StringView |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| uuid | FIXED_LEN_BYTE_ARRAY[len=16], UUID | :ref:`UUID extension type <uuid_extension>` |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+
+The decimal precision bands follow the `Variant encoding types
Review Comment:
I am still a little confused about how decimals are working -- is the this text talking about each variant value itself (as the scale/precision can vary row to row) or the min/max precision of all values in a column (presumably what is needed when shredding) 😕
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3804929724
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,82 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It is derived from the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification: each row maps a Variant
+primitive type to the Parquet type required for a shredded ``typed_value``
+column (physical type, followed by the logical type annotation if any) and to
+the Arrow type(s) able to represent that Variant type's full value domain.
+A ``typed_value`` field of one of the listed Arrow types holds values of
+exactly the corresponding Variant type, and the listed Parquet type is its
+only valid Parquet representation.
+
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| Variant Type | Parquet Type | Arrow ``typed_value`` Type |
++========================================+==================================================+=============================================+
+| boolean | BOOLEAN | Boolean |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int8 | INT32, INT(8, true) | Int8 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int16 | INT32, INT(16, true) | Int16 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int32 | INT32 | Int32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int64 | INT64 | Int64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| float | FLOAT | Float32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| double | DOUBLE | Float64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32, DECIMAL(P, S) | Decimal32(P, S) |
Review Comment:
There's a small separate table for decimals that shows what precision can fit in the underlying physical type.
https://github.com/apache/parquet-format/blob/3ef43e00cea6dcd06da5b77541f84bef879aff5b/VariantEncoding.md?plain=1#L440-L447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3926985275
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,107 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It follows the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification. Each row maps a Variant
+primitive type to the Parquet `physical type
+<https://parquet.apache.org/docs/file-format/types/>`__ and `logical type
+<https://parquet.apache.org/docs/file-format/types/logicaltypes/>`__
+annotation of a shredded ``typed_value`` column, and to the Arrow
+:ref:`data types <data_types>` covering that Variant type's full value
+domain. An empty *Parquet Logical Type* cell means the physical type carries
+no explicit annotation. This is intentional for ``int32`` and ``int64``:
+Parquet defines ``INT(32, true)`` and ``INT(64, true)`` as implied by the
+unannotated ``INT32`` and ``INT64`` physical types, respectively.
Review Comment:
I added it due to @alamb comment above
> I am a little confused about what (logical or physical) type is used in the Parquet Type column. Some seem to be just the physical type (INT64) , and some seem to have both logical and physical (INT32, INT(8, true) ). It would help to be spcific about what the Parquet type column represents and use it consistently
The Variant spec omits logical types IN32, INT64 the same way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #50810:
URL: https://github.com/apache/arrow/pull/50810#issuecomment-5331064421
@alamb I think it's more clear now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3927648096
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,107 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It follows the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification. Each row maps a Variant
+primitive type to the Parquet `physical type
+<https://parquet.apache.org/docs/file-format/types/>`__ and `logical type
+<https://parquet.apache.org/docs/file-format/types/logicaltypes/>`__
+annotation of a shredded ``typed_value`` column, and to the Arrow
+:ref:`data types <data_types>` covering that Variant type's full value
+domain. An empty *Parquet Logical Type* cell means the physical type carries
+no explicit annotation. This is intentional for ``int32`` and ``int64``:
+Parquet defines ``INT(32, true)`` and ``INT(64, true)`` as implied by the
+unannotated ``INT32`` and ``INT64`` physical types, respectively.
+
+A ``typed_value`` field of a listed Arrow type holds values of exactly the
+corresponding Variant type, and the listed physical and logical types are its
+only valid Parquet representation.
+
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| Variant Type | Parquet Physical Type | Parquet Logical Type | Arrow ``typed_value`` Type |
++========================================+===================================+==========================+=============================================+
+| boolean | BOOLEAN | | Boolean |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int8 | INT32 | INT(8, true) | Int8 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int16 | INT32 | INT(16, true) | Int16 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int32 | INT32 | | Int32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int64 | INT64 | | Int64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| float | FLOAT | | Float32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| double | DOUBLE | | Float64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32 | DECIMAL(P, S) | Decimal32(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal8 (10 <= P <= 18, 0 <= S <= P) | INT64 | DECIMAL(P, S) | Decimal64(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal16 (19 <= P <= 38, 0 <= S <= P) | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY | DECIMAL(P, S) | Decimal128(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| date | INT32 | DATE | Date32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| time | INT64 | TIME(false, MICROS) | Time64(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(6) | INT64 | TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(9) | INT64 | TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(6) | INT64 | TIMESTAMP(false, MICROS) | Timestamp(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(9) | INT64 | TIMESTAMP(false, NANOS) | Timestamp(ns) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| binary | BYTE_ARRAY | | Binary / LargeBinary / BinaryView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| string | BYTE_ARRAY | STRING | String / LargeString / StringView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| uuid | FIXED_LEN_BYTE_ARRAY[len=16] | UUID | :ref:`UUID extension type <uuid_extension>` |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+
+The decimal precision bands follow the *Decimal table* of the `Variant
+encoding specification
+<https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types>`__
+(just below its *Encoding types* table). The bands are disjoint, so
+precision alone selects the row (the narrowest sufficient decimal type is
+required) and the scale must satisfy ``0 <= S <= P``. Arrow decimal types
+outside these bounds (a negative scale, or a wider decimal type than the
+precision requires) are not valid ``typed_value`` storage.
+
+.. note::
+
+ The bands reflect the underlying physical type: a Variant decimal
+ stores its unscaled value as a 4-, 8-, or 16-byte integer, capping the
+ precision at 9, 18, or 38 significant digits.
+
+ ``(P, S)`` refers to the Arrow data type of the ``typed_value`` field
Review Comment:
per @alamb's comment above it seems like it was not clean enough before
> I am still a little confused about how decimals are working -- is the this text talking about each variant value itself (as the scale/precision can vary row to row) or the min/max precision of all values in a column (presumably what is needed when shredding) 😕
I can reword it if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #50810:
URL: https://github.com/apache/arrow/pull/50810#issuecomment-5374148943
Thanks @alamb addressed your comments here - https://github.com/apache/arrow/pull/50810/commits/84af803631a593f191aa21b8241eecbe597d4faf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #50810:
URL: https://github.com/apache/arrow/pull/50810#issuecomment-5684401052
Re-requested the reviews.
I no one has objections I'd like to merge by the end of the week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3808480831
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,102 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It follows the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification. Each row maps a Variant
+primitive type to the Parquet `physical type
+<https://parquet.apache.org/docs/file-format/types/>`__ and `logical type
+<https://parquet.apache.org/docs/file-format/types/logicaltypes/>`__
+annotation of a shredded ``typed_value`` column, and to the Arrow
+:ref:`data types <data_types>` covering that Variant type's full value
+domain. An empty *Parquet Logical Type* cell means the physical type carries
+no annotation.
+
+A ``typed_value`` field of a listed Arrow type holds values of exactly the
+corresponding Variant type, and the listed physical and logical types are its
+only valid Parquet representation.
+
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| Variant Type | Parquet Physical Type | Parquet Logical Type | Arrow ``typed_value`` Type |
++========================================+===================================+==========================+=============================================+
+| boolean | BOOLEAN | | Boolean |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int8 | INT32 | INT(8, true) | Int8 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int16 | INT32 | INT(16, true) | Int16 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int32 | INT32 | | Int32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int64 | INT64 | | Int64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| float | FLOAT | | Float32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| double | DOUBLE | | Float64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32 | DECIMAL(P, S) | Decimal32(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal8 (10 <= P <= 18, 0 <= S <= P) | INT64 | DECIMAL(P, S) | Decimal64(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal16 (19 <= P <= 38, 0 <= S <= P) | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY | DECIMAL(P, S) | Decimal128(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| date | INT32 | DATE | Date32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| time | INT64 | TIME(false, MICROS) | Time64(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(6) | INT64 | TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(9) | INT64 | TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(6) | INT64 | TIMESTAMP(false, MICROS) | Timestamp(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(9) | INT64 | TIMESTAMP(false, NANOS) | Timestamp(ns) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| binary | BYTE_ARRAY | | Binary / LargeBinary / BinaryView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| string | BYTE_ARRAY | STRING | String / LargeString / StringView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| uuid | FIXED_LEN_BYTE_ARRAY[len=16] | UUID | :ref:`UUID extension type <uuid_extension>` |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+
+The decimal precision bands follow the *Decimal table* of the `Variant
+encoding specification
+<https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types>`__
+(just below its *Encoding types* table). The bands are disjoint, so
+precision alone selects the row (the narrowest sufficient decimal type is
+required) and the scale must satisfy ``0 <= S <= P``. Arrow decimal types
+outside these bounds (a negative scale, or a wider decimal type than the
+precision requires) are not valid ``typed_value`` storage.
+
+.. note::
+
+ The bands reflect the underlying physical type: a Variant decimal
+ stores its unscaled value as a 4-, 8-, or 16-byte integer, capping the
+ precision at 9, 18, or 38 significant digits.
+
+ ``(P, S)`` refers to the Arrow data type of the ``typed_value`` field
+ (``Decimal32(7, 2)``, for example) and thus applies to the array as a
+ whole: an encoded decimal value carries only a scale (its precision is
+ implied by the unscaled value), while the field's type fixes one
+ precision and scale for every row. A value that the field's type cannot
+ represent is not shredded and remains Variant-encoded in the ``value``
+ field.
+
+.. note::
+
+ Arrow types without a row in this table (such as ``Null`` or the unsigned
+ integer types) must not be used as ``typed_value`` storage, as they have no
+ valid Parquet shredded representation:
+
+ * A Variant null is always encoded in the ``value`` field (as ``00``),
+ never in ``typed_value``: a null ``typed_value`` signals that the row is
+ not shredded, and for shredded object fields a null ``typed_value``
+ together with a null ``value`` means the field is missing.
Review Comment:
I don't think it should.
The spec explicitly says:
> * The `value` field must be annotated as `required` for unshredded Variant values, or `optional` if parts of the value are [shredded](VariantShredding.md) as typed Parquet columns.
[link](https://github.com/apache/parquet-format/blob/3ef43e00cea6dcd06da5b77541f84bef879aff5b/VariantEncoding.md?plain=1#L50)
That makes `Variant::Null` and SQL NULL different things
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3807481886
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,102 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It follows the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification. Each row maps a Variant
+primitive type to the Parquet `physical type
+<https://parquet.apache.org/docs/file-format/types/>`__ and `logical type
+<https://parquet.apache.org/docs/file-format/types/logicaltypes/>`__
+annotation of a shredded ``typed_value`` column, and to the Arrow
+:ref:`data types <data_types>` covering that Variant type's full value
+domain. An empty *Parquet Logical Type* cell means the physical type carries
+no annotation.
+
+A ``typed_value`` field of a listed Arrow type holds values of exactly the
+corresponding Variant type, and the listed physical and logical types are its
+only valid Parquet representation.
+
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| Variant Type | Parquet Physical Type | Parquet Logical Type | Arrow ``typed_value`` Type |
++========================================+===================================+==========================+=============================================+
+| boolean | BOOLEAN | | Boolean |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int8 | INT32 | INT(8, true) | Int8 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int16 | INT32 | INT(16, true) | Int16 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int32 | INT32 | | Int32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int64 | INT64 | | Int64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| float | FLOAT | | Float32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| double | DOUBLE | | Float64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32 | DECIMAL(P, S) | Decimal32(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal8 (10 <= P <= 18, 0 <= S <= P) | INT64 | DECIMAL(P, S) | Decimal64(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal16 (19 <= P <= 38, 0 <= S <= P) | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY | DECIMAL(P, S) | Decimal128(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| date | INT32 | DATE | Date32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| time | INT64 | TIME(false, MICROS) | Time64(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(6) | INT64 | TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(9) | INT64 | TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(6) | INT64 | TIMESTAMP(false, MICROS) | Timestamp(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(9) | INT64 | TIMESTAMP(false, NANOS) | Timestamp(ns) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| binary | BYTE_ARRAY | | Binary / LargeBinary / BinaryView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| string | BYTE_ARRAY | STRING | String / LargeString / StringView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| uuid | FIXED_LEN_BYTE_ARRAY[len=16] | UUID | :ref:`UUID extension type <uuid_extension>` |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+
+The decimal precision bands follow the *Decimal table* of the `Variant
+encoding specification
+<https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types>`__
+(just below its *Encoding types* table). The bands are disjoint, so
+precision alone selects the row (the narrowest sufficient decimal type is
+required) and the scale must satisfy ``0 <= S <= P``. Arrow decimal types
+outside these bounds (a negative scale, or a wider decimal type than the
+precision requires) are not valid ``typed_value`` storage.
+
+.. note::
+
+ The bands reflect the underlying physical type: a Variant decimal
+ stores its unscaled value as a 4-, 8-, or 16-byte integer, capping the
+ precision at 9, 18, or 38 significant digits.
+
+ ``(P, S)`` refers to the Arrow data type of the ``typed_value`` field
+ (``Decimal32(7, 2)``, for example) and thus applies to the array as a
+ whole: an encoded decimal value carries only a scale (its precision is
+ implied by the unscaled value), while the field's type fixes one
+ precision and scale for every row. A value that the field's type cannot
+ represent is not shredded and remains Variant-encoded in the ``value``
+ field.
+
+.. note::
+
+ Arrow types without a row in this table (such as ``Null`` or the unsigned
+ integer types) must not be used as ``typed_value`` storage, as they have no
+ valid Parquet shredded representation:
+
+ * A Variant null is always encoded in the ``value`` field (as ``00``),
+ never in ``typed_value``: a null ``typed_value`` signals that the row is
+ not shredded, and for shredded object fields a null ``typed_value``
+ together with a null ``value`` means the field is missing.
Review Comment:
Should we clarify that a variant null should still be indicated in the null validity bitmap of the corresponding array (accompanying the `00` value in the `value` field)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3812628000
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,102 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It follows the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification. Each row maps a Variant
+primitive type to the Parquet `physical type
+<https://parquet.apache.org/docs/file-format/types/>`__ and `logical type
+<https://parquet.apache.org/docs/file-format/types/logicaltypes/>`__
+annotation of a shredded ``typed_value`` column, and to the Arrow
+:ref:`data types <data_types>` covering that Variant type's full value
+domain. An empty *Parquet Logical Type* cell means the physical type carries
+no annotation.
+
+A ``typed_value`` field of a listed Arrow type holds values of exactly the
+corresponding Variant type, and the listed physical and logical types are its
+only valid Parquet representation.
+
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| Variant Type | Parquet Physical Type | Parquet Logical Type | Arrow ``typed_value`` Type |
++========================================+===================================+==========================+=============================================+
+| boolean | BOOLEAN | | Boolean |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int8 | INT32 | INT(8, true) | Int8 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int16 | INT32 | INT(16, true) | Int16 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int32 | INT32 | | Int32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int64 | INT64 | | Int64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| float | FLOAT | | Float32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| double | DOUBLE | | Float64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32 | DECIMAL(P, S) | Decimal32(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal8 (10 <= P <= 18, 0 <= S <= P) | INT64 | DECIMAL(P, S) | Decimal64(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal16 (19 <= P <= 38, 0 <= S <= P) | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY | DECIMAL(P, S) | Decimal128(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| date | INT32 | DATE | Date32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| time | INT64 | TIME(false, MICROS) | Time64(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(6) | INT64 | TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(9) | INT64 | TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(6) | INT64 | TIMESTAMP(false, MICROS) | Timestamp(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(9) | INT64 | TIMESTAMP(false, NANOS) | Timestamp(ns) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| binary | BYTE_ARRAY | | Binary / LargeBinary / BinaryView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| string | BYTE_ARRAY | STRING | String / LargeString / StringView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| uuid | FIXED_LEN_BYTE_ARRAY[len=16] | UUID | :ref:`UUID extension type <uuid_extension>` |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+
+The decimal precision bands follow the *Decimal table* of the `Variant
+encoding specification
+<https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types>`__
+(just below its *Encoding types* table). The bands are disjoint, so
+precision alone selects the row (the narrowest sufficient decimal type is
+required) and the scale must satisfy ``0 <= S <= P``. Arrow decimal types
+outside these bounds (a negative scale, or a wider decimal type than the
+precision requires) are not valid ``typed_value`` storage.
+
+.. note::
+
+ The bands reflect the underlying physical type: a Variant decimal
+ stores its unscaled value as a 4-, 8-, or 16-byte integer, capping the
+ precision at 9, 18, or 38 significant digits.
+
+ ``(P, S)`` refers to the Arrow data type of the ``typed_value`` field
+ (``Decimal32(7, 2)``, for example) and thus applies to the array as a
+ whole: an encoded decimal value carries only a scale (its precision is
+ implied by the unscaled value), while the field's type fixes one
+ precision and scale for every row. A value that the field's type cannot
+ represent is not shredded and remains Variant-encoded in the ``value``
+ field.
+
+.. note::
+
+ Arrow types without a row in this table (such as ``Null`` or the unsigned
+ integer types) must not be used as ``typed_value`` storage, as they have no
+ valid Parquet shredded representation:
+
+ * A Variant null is always encoded in the ``value`` field (as ``00``),
+ never in ``typed_value``: a null ``typed_value`` signals that the row is
+ not shredded, and for shredded object fields a null ``typed_value``
+ together with a null ``value`` means the field is missing.
Review Comment:
I think it might be more precise to say it makes `Variant::Null` and a null value in the corresponding arrow array different things
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,102 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It follows the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification. Each row maps a Variant
+primitive type to the Parquet `physical type
+<https://parquet.apache.org/docs/file-format/types/>`__ and `logical type
+<https://parquet.apache.org/docs/file-format/types/logicaltypes/>`__
+annotation of a shredded ``typed_value`` column, and to the Arrow
+:ref:`data types <data_types>` covering that Variant type's full value
+domain. An empty *Parquet Logical Type* cell means the physical type carries
+no annotation.
+
+A ``typed_value`` field of a listed Arrow type holds values of exactly the
+corresponding Variant type, and the listed physical and logical types are its
+only valid Parquet representation.
+
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| Variant Type | Parquet Physical Type | Parquet Logical Type | Arrow ``typed_value`` Type |
++========================================+===================================+==========================+=============================================+
+| boolean | BOOLEAN | | Boolean |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int8 | INT32 | INT(8, true) | Int8 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int16 | INT32 | INT(16, true) | Int16 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int32 | INT32 | | Int32 |
Review Comment:
i am confused that there is no logical type for int32, though this is the same in the parquet variant table too https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3921399158
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,107 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It follows the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification. Each row maps a Variant
+primitive type to the Parquet `physical type
+<https://parquet.apache.org/docs/file-format/types/>`__ and `logical type
+<https://parquet.apache.org/docs/file-format/types/logicaltypes/>`__
+annotation of a shredded ``typed_value`` column, and to the Arrow
+:ref:`data types <data_types>` covering that Variant type's full value
+domain. An empty *Parquet Logical Type* cell means the physical type carries
+no explicit annotation. This is intentional for ``int32`` and ``int64``:
+Parquet defines ``INT(32, true)`` and ``INT(64, true)`` as implied by the
+unannotated ``INT32`` and ``INT64`` physical types, respectively.
Review Comment:
```suggestion
no explicit annotation.
```
Do we really need this since it is already covered by the Parquet Variant spec?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,107 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It follows the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification. Each row maps a Variant
+primitive type to the Parquet `physical type
+<https://parquet.apache.org/docs/file-format/types/>`__ and `logical type
+<https://parquet.apache.org/docs/file-format/types/logicaltypes/>`__
+annotation of a shredded ``typed_value`` column, and to the Arrow
+:ref:`data types <data_types>` covering that Variant type's full value
+domain. An empty *Parquet Logical Type* cell means the physical type carries
+no explicit annotation. This is intentional for ``int32`` and ``int64``:
+Parquet defines ``INT(32, true)`` and ``INT(64, true)`` as implied by the
+unannotated ``INT32`` and ``INT64`` physical types, respectively.
+
+A ``typed_value`` field of a listed Arrow type holds values of exactly the
+corresponding Variant type, and the listed physical and logical types are its
+only valid Parquet representation.
+
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| Variant Type | Parquet Physical Type | Parquet Logical Type | Arrow ``typed_value`` Type |
++========================================+===================================+==========================+=============================================+
+| boolean | BOOLEAN | | Boolean |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int8 | INT32 | INT(8, true) | Int8 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int16 | INT32 | INT(16, true) | Int16 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int32 | INT32 | | Int32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| int64 | INT64 | | Int64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| float | FLOAT | | Float32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| double | DOUBLE | | Float64 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32 | DECIMAL(P, S) | Decimal32(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal8 (10 <= P <= 18, 0 <= S <= P) | INT64 | DECIMAL(P, S) | Decimal64(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| decimal16 (19 <= P <= 38, 0 <= S <= P) | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY | DECIMAL(P, S) | Decimal128(P, S) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| date | INT32 | DATE | Date32 |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| time | INT64 | TIME(false, MICROS) | Time64(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(6) | INT64 | TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestamptz(9) | INT64 | TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(6) | INT64 | TIMESTAMP(false, MICROS) | Timestamp(us) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| timestampntz(9) | INT64 | TIMESTAMP(false, NANOS) | Timestamp(ns) |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| binary | BYTE_ARRAY | | Binary / LargeBinary / BinaryView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| string | BYTE_ARRAY | STRING | String / LargeString / StringView |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+| uuid | FIXED_LEN_BYTE_ARRAY[len=16] | UUID | :ref:`UUID extension type <uuid_extension>` |
++----------------------------------------+-----------------------------------+--------------------------+---------------------------------------------+
+
+The decimal precision bands follow the *Decimal table* of the `Variant
+encoding specification
+<https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types>`__
+(just below its *Encoding types* table). The bands are disjoint, so
+precision alone selects the row (the narrowest sufficient decimal type is
+required) and the scale must satisfy ``0 <= S <= P``. Arrow decimal types
+outside these bounds (a negative scale, or a wider decimal type than the
+precision requires) are not valid ``typed_value`` storage.
+
+.. note::
+
+ The bands reflect the underlying physical type: a Variant decimal
+ stores its unscaled value as a 4-, 8-, or 16-byte integer, capping the
+ precision at 9, 18, or 38 significant digits.
+
+ ``(P, S)`` refers to the Arrow data type of the ``typed_value`` field
Review Comment:
Do we really need to re-explain what a standard Arrow Decimal Array is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #50810:
URL: https://github.com/apache/arrow/pull/50810#issuecomment-5282519965
Please take a look @alamb @wgtmac @zeroshade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #50810:
URL: https://github.com/apache/arrow/pull/50810#issuecomment-5758956511
Thanks again everyone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50810:
URL: https://github.com/apache/arrow/pull/50810#issuecomment-5184433014
:warning: GitHub issue #50622 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on PR #50810:
URL: https://github.com/apache/arrow/pull/50810#issuecomment-5401446220
@alamb I think my message to the mailing list about this went to spam too :cry:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3804929724
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,82 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It is derived from the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification: each row maps a Variant
+primitive type to the Parquet type required for a shredded ``typed_value``
+column (physical type, followed by the logical type annotation if any) and to
+the Arrow type(s) able to represent that Variant type's full value domain.
+A ``typed_value`` field of one of the listed Arrow types holds values of
+exactly the corresponding Variant type, and the listed Parquet type is its
+only valid Parquet representation.
+
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| Variant Type | Parquet Type | Arrow ``typed_value`` Type |
++========================================+==================================================+=============================================+
+| boolean | BOOLEAN | Boolean |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int8 | INT32, INT(8, true) | Int8 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int16 | INT32, INT(16, true) | Int16 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int32 | INT32 | Int32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int64 | INT64 | Int64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| float | FLOAT | Float32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| double | DOUBLE | Float64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32, DECIMAL(P, S) | Decimal32(P, S) |
Review Comment:
There's a small separate table for decimals that shows what precision can fit in the underlying physical type of a `typed_value`.
https://github.com/apache/parquet-format/blob/3ef43e00cea6dcd06da5b77541f84bef879aff5b/VariantEncoding.md?plain=1#L440-L447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #50810:
URL: https://github.com/apache/arrow/pull/50810#issuecomment-5704191217
I'll plan to merge this on Friday Sep 18 2026 unless anyone objects or would like more time to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #50810:
URL: https://github.com/apache/arrow/pull/50810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "HuaHuaY (via GitHub)" <gi...@apache.org>.
HuaHuaY commented on PR #50810:
URL: https://github.com/apache/arrow/pull/50810#issuecomment-5404279420
> @alamb I think my message to the mailing list about this went to spam too 😢
I received the email successfully. I guess the community people may be busy lately. Perhaps you could send another email to remind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50622: [Docs][Format] Align Variant `typed_value` primitive type mappings with the Parquet shredding spec [arrow]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on code in PR #50810:
URL: https://github.com/apache/arrow/pull/50810#discussion_r3805097792
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -488,63 +488,82 @@ binary values look like.
Primitive Type Mappings
-----------------------
-+----------------------+------------------------+
-| Arrow Primitive Type | Variant Primitive Type |
-+======================+========================+
-| Null | Null |
-+----------------------+------------------------+
-| Boolean | Boolean (true/false) |
-+----------------------+------------------------+
-| Int8 | Int8 |
-+----------------------+------------------------+
-| Uint8 | Int16 |
-+----------------------+------------------------+
-| Int16 | Int16 |
-+----------------------+------------------------+
-| Uint16 | Int32 |
-+----------------------+------------------------+
-| Int32 | Int32 |
-+----------------------+------------------------+
-| Uint32 | Int64 |
-+----------------------+------------------------+
-| Int64 | Int64 |
-+----------------------+------------------------+
-| Float | Float |
-+----------------------+------------------------+
-| Double | Double |
-+----------------------+------------------------+
-| Decimal32 | decimal4 |
-+----------------------+------------------------+
-| Decimal64 | decimal8 |
-+----------------------+------------------------+
-| Decimal128 | decimal16 |
-+----------------------+------------------------+
-| Date32 | Date |
-+----------------------+------------------------+
-| Time64 | TimeNTZ |
-+----------------------+------------------------+
-| Timestamp(us, UTC) | Timestamp (micro) |
-+----------------------+------------------------+
-| Timestamp(us) | TimestampNTZ (micro) |
-+----------------------+------------------------+
-| Timestamp(ns, UTC) | Timestamp (nano) |
-+----------------------+------------------------+
-| Timestamp(ns) | TimestampNTZ (nano) |
-+----------------------+------------------------+
-| Binary | Binary |
-+----------------------+------------------------+
-| LargeBinary | Binary |
-+----------------------+------------------------+
-| BinaryView | Binary |
-+----------------------+------------------------+
-| String | String |
-+----------------------+------------------------+
-| LargeString | String |
-+----------------------+------------------------+
-| StringView | String |
-+----------------------+------------------------+
-| UUID extension type | UUID |
-+----------------------+------------------------+
+The following table defines the set of Arrow types that are valid as primitive
+``typed_value`` storage. It is derived from the `Shredded Value Types
+<https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types>`__
+table of the Parquet Variant Shredding specification: each row maps a Variant
+primitive type to the Parquet type required for a shredded ``typed_value``
+column (physical type, followed by the logical type annotation if any) and to
+the Arrow type(s) able to represent that Variant type's full value domain.
+A ``typed_value`` field of one of the listed Arrow types holds values of
+exactly the corresponding Variant type, and the listed Parquet type is its
+only valid Parquet representation.
+
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| Variant Type | Parquet Type | Arrow ``typed_value`` Type |
++========================================+==================================================+=============================================+
+| boolean | BOOLEAN | Boolean |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int8 | INT32, INT(8, true) | Int8 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int16 | INT32, INT(16, true) | Int16 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int32 | INT32 | Int32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| int64 | INT64 | Int64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| float | FLOAT | Float32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| double | DOUBLE | Float64 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal4 (1 <= P <= 9, 0 <= S <= P) | INT32, DECIMAL(P, S) | Decimal32(P, S) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal8 (10 <= P <= 18, 0 <= S <= P) | INT64, DECIMAL(P, S) | Decimal64(P, S) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| decimal16 (19 <= P <= 38, 0 <= S <= P) | BYTE_ARRAY / FIXED_LEN_BYTE_ARRAY, DECIMAL(P, S) | Decimal128(P, S) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| date | INT32, DATE | Date32 |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| time | INT64, TIME(false, MICROS) | Time64(us) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| timestamptz(6) | INT64, TIMESTAMP(true, MICROS) | Timestamp(us, UTC) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| timestamptz(9) | INT64, TIMESTAMP(true, NANOS) | Timestamp(ns, UTC) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| timestampntz(6) | INT64, TIMESTAMP(false, MICROS) | Timestamp(us) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| timestampntz(9) | INT64, TIMESTAMP(false, NANOS) | Timestamp(ns) |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| binary | BYTE_ARRAY | Binary / LargeBinary / BinaryView |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| string | BYTE_ARRAY, STRING | String / LargeString / StringView |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+| uuid | FIXED_LEN_BYTE_ARRAY[len=16], UUID | :ref:`UUID extension type <uuid_extension>` |
++----------------------------------------+--------------------------------------------------+---------------------------------------------+
+
+The decimal precision bands follow the `Variant encoding types
Review Comment:
It's talking about the constraint of physical types of the `typed_value` field, so it would be the latter. I'll polish it to make it more clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org