You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zeroshade (via GitHub)" <gi...@apache.org> on 2025/08/28 19:34:13 UTC
[PR] GH-46908: add variant extension type docs [arrow]
zeroshade opened a new pull request, #47456:
URL: https://github.com/apache/arrow/pull/47456
### Rationale for this change
To support the addition of the Parquet Variant data type and the Iceberg adoption of the variant type, we need a defined way to pass this data through Arrow-compatible systems. As such, we need a specification for a canonical Arrow extension type to represent Variant data.
### What changes are included in this PR?
Updates to the docs which define the Arrow Variant Extension type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2318100646
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
Review Comment:
We can add a later parameter to denote future versions, if any?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316283784
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
Review Comment:
```suggestion
* If the ``typed_value`` field is a *nested* type, it **must** be a non-nullable ``Struct`` consisting of
at least one (or both) of the following:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316426200
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
Review Comment:
I'm not sure if examples of data layout are required because these are not new knowledge to readers. However, it might be useful for developers who implement the Variant type support to get started with these examples?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2318100646
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
Review Comment:
We can add a later optional parameter to denote future versions of the Variant spec, if any?
(other Parquet spec additions would not matter)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316375922
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
Review Comment:
That was your suggestion on the google doc proposal :smile: I'll change the wording a bit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319886014
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
Review Comment:
It sounds like applications don't _have_ to reorder but may choose to and would need to be aware that they must access fields by name and not position. 1:1 compatibility with what Parquet is doing sounds like the better tradeoff (Option 2 above).
Could we extend the existing Note about field order or add another Note succinctly explaining this? i.e., answer @emkornfield question in the final document.
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,592 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _parquet_variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
Review Comment:
This isn't rendering correctly,
<img width="1274" height="178" alt="Image" src="https://github.com/user-attachments/assets/4149a320-0fb0-4752-a6ef-cff2b1a739f1" />
I suggest changing to:
```suggestion
(unshredded variants consist of just the ``metadata`` and ``value`` fields only)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319831354
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
Review Comment:
Since the primary stated purpose of this extension type is to provide near-zero-copy compatibility with Parquet Variant, and since we expect that its dominant use will be for efficient transfer to or from Parquet, I think it makes sense to defer the complexity and performance overheads required for other uses. This suggests to me that Option 2 is preferable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: add variant extension type docs [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3234710187
:warning: GitHub issue #46908 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319910055
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _parquet_variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |-------------------------|--------------------------|
+ | 01 00 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
Review Comment:
How is it now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2317515710
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
Review Comment:
i think this covers my question above about dictionary, but not run-end encoded.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319025401
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
Review Comment:
If we go with `parquet.` we should also change [this note](https://github.com/apache/arrow/blob/8ccdbe78063ad4b43872b8826aba37a1a73dc951/docs/source/format/Columnar.rst?plain=1#L1660-L1663) to specify that it's reserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2335746738
##########
docs/source/format/CanonicalExtensions/Examples.rst:
##########
@@ -0,0 +1,555 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_canonical_extension_examples:
+
+****************************
+Canonical Extension Examples
+****************************
+
+=========================
+Parquet Variant Extension
+=========================
+
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``arrow.parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: int64 nullable
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
Review Comment:
@zeroshade Thanks for the reply!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316283784
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
Review Comment:
```suggestion
* If the ``typed_value`` field is a *nested* type, **must** be a non-nullable ``Struct`` consisting of
at least one (or both) of the following:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3267411419
Once we have consensus and approval here, we can merge this. Can everyone please take a look and try to comment on this (or give approval) by the end of the week?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2331071917
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,133 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _parquet_variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``arrow.parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ (unshredded variants consist of just the ``metadata`` and ``value`` fields only)
+
+ * A field named ``typed_value`` which can be a :ref:`variant_primitive_type_mapping` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a ``List``, ``LargeList`` or ``ListView`` its elements **must** be *non-nullable* and **must**
+ be a ``Struct`` consisting of at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+
+ * If the ``typed_value`` field is a ``Struct``, then its fields **must** be *non-nullable*, representing the fields being shredded
+ from the objects, and **must** be a ``Struct`` consisting of at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``,
+ or run-end-encoded with a preferred (*but not required*) runs type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+.. _variant_primitive_type_mapping:
+
+Primitive Type Mappings
+-----------------------
+
++----------------------+------------------------+
+| Arrow Primitive Type | Variant Primitive Type |
++======================+========================+
+| Null | Null |
++----------------------+------------------------+
+| Boolean (true/false) | Boolean |
Review Comment:
I wonder if (true/false) is meant to apply to the other column in the variant primitive type (which has separate primitive types for `true` and `false`)?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,133 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _parquet_variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
Review Comment:
Here is a suggestion to clarify and make this more concise (I think it is also fine as is)
```suggestion
variant values. The canonical extension type allows systems to pass Variant encoded data around without special handling unless they want to directly interact with the encoded variant data. See the Parquet format specification for details on what the actual binary values look like.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3268477427
Revision: 44da918127c4e737785e9fcc9be99512b50726cf
Submitted crossbow builds: [ursacomputing/crossbow @ actions-fdb42d9677](https://github.com/ursacomputing/crossbow/branches/all?query=actions-fdb42d9677)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/17568350172/job/49899621503)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #47456:
URL: https://github.com/apache/arrow/pull/47456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2330938569
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
Review Comment:
I agree that `arrow.parquet` for this spec sounds good (and thus revert the changes to this section of the spec)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2331124199
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,133 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _parquet_variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``arrow.parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ (unshredded variants consist of just the ``metadata`` and ``value`` fields only)
+
+ * A field named ``typed_value`` which can be a :ref:`variant_primitive_type_mapping` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a ``List``, ``LargeList`` or ``ListView`` its elements **must** be *non-nullable* and **must**
+ be a ``Struct`` consisting of at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+
+ * If the ``typed_value`` field is a ``Struct``, then its fields **must** be *non-nullable*, representing the fields being shredded
+ from the objects, and **must** be a ``Struct`` consisting of at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``,
+ or run-end-encoded with a preferred (*but not required*) runs type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+.. _variant_primitive_type_mapping:
+
+Primitive Type Mappings
+-----------------------
+
++----------------------+------------------------+
+| Arrow Primitive Type | Variant Primitive Type |
++======================+========================+
+| Null | Null |
++----------------------+------------------------+
+| Boolean (true/false) | Boolean |
Review Comment:
good point, my mistake there. I'll swap these
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3268787128
Preview: http://crossbow.voltrondata.com/pr_docs/47456/format/CanonicalExtensions.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316296807
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
Review Comment:
Do we need the examples to be that long-winded? For the most part, we can refer to the Parquet specification, IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316278914
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
Review Comment:
```suggestion
* Extension type parameters:
This type does not have any parameters.
* Description of the serialization:
Extension metadata is an empty string.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316385353
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
Review Comment:
a struct of non-nullable structs.
It's a struct of non-nullable structs for the same reason it is a List/LargeList/ListView of non-nullable structs: because each needs to be a pair of `value` and `typed_value` (which follows the parquet Variant shredding spec)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316417416
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
Review Comment:
The fields of the outer struct are the fields being shredded, see the `Shredding an Object` example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319710678
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
Review Comment:
I think run-end-encoding should be fine personally. Anyone object to updating this to say that REE is also permissible? @wgtmac @pitrou @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2318100646
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
Review Comment:
We can add a later optional parameter to denote future versions, if any?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2331034677
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
Review Comment:
I will revert this change, since it's no longer needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319729955
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
Review Comment:
In a discussion about this in the Parquet sync meeting today, there was a general preference for `arrow.parquet.` because it does not expand the previously reserved namespaces, avoids future possible name collisions. It sounds like we'll go with that unless there are well-reasoned objections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2310310371
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
Review Comment:
Sure, I'll update all the statements to be `nullable/non-nullable` instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316278914
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
Review Comment:
```suggestion
* Extension type parameters:
This type does not have any parameters.
* Description of the serialization:
Metadata is an empty string.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316292424
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
Review Comment:
```suggestion
* A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316305234
##########
docs/source/status.rst:
##########
@@ -129,6 +129,8 @@ Data Types
+-----------------------+-------+-------+-------+------------+-------+-------+-------+-------+
| 8-bit Boolean | ✓ | | ✓ | | | | | |
+-----------------------+-------+-------+-------+------------+-------+-------+-------+-------+
+| Variant | | | ✓ | | | | | |
Review Comment:
Let's name it "Parquet Variant" explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316400805
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
Review Comment:
Ok, the inner struct has `value` and `typed_value` fields, but what fields does the outer struct have?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2317508715
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
Review Comment:
nit: I think `arrow.parquet` might avoid a conflict but this is potentially minor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2318107124
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
Review Comment:
Probably because it mirrors what was decided on the Parquet side. However, there's a trade-off here:
1) if we are prescriptive on ordering, Parquet readers have to reorder fields when converting from Parquet columns if necessary, and applications can assume a given order
2) if we are not prescriptive on ordering, applications have to reorder
Option 1 sounds slightly better to me (and readers will have to reorder at some point if they grow higher-level variant access APIs anyway).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2332360353
##########
docs/source/format/CanonicalExtensions/Examples.rst:
##########
@@ -0,0 +1,555 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_canonical_extension_examples:
+
+****************************
+Canonical Extension Examples
+****************************
+
+=========================
+Parquet Variant Extension
+=========================
+
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``arrow.parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: int64 nullable
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
Review Comment:
`01`(byte[0]) indicate the header with `offset_size_minus_one` = 0, and `00`(byte[1]) means the `dictionary_size` = 0, so there would no `offset` list and `bytes` following.
There are four `01 00` in the meta, because we have four `variant`/row(`34, null, "n/a", 100`) in all here, am I understand right? If it's yes, does this mean we'll have ${lenght_of_row} pair of `<header,dictionay_size, offset, bytes>` in the metadata `VarBinary`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2317515238
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
Review Comment:
the parquet spec is versioned for forward compatiblity, are we going to ignore that for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2317513808
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
Review Comment:
these can be dictionary encoded and run-end encoded as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316303067
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
Review Comment:
Or, if we really want the long form examples, it would be better IMHO to move them to a separate document so that the canonical extensions page remains easily readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316280727
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
Review Comment:
Why only int8?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316402991
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
Review Comment:
That said, I can see that they can be useful for implementors or to improve understanding. So my vote would be now on a separate document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: add variant extension type docs [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2308951039
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
Review Comment:
Would nullable/non-nullable be more arrowish (required/optional I think is still Parquet?)
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of: ::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field.
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like: ::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*. A null
+element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+The storage type to represent this in Arrow as a Variant extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: list<element: struct<
+ value: binary optional,
+ typed_value: string optional
+ > required> optional
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
+
+The data would then be stored in Arrow as follows: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000111 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary` Metadata)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary` Value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 0, 0, 1 | unspecified (padding) |
+
+ * Value buffer: (00 -> variant null)
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-2 array (`List<Struct<VarBinary, String>>` typed_value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00000111 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |-------------------|-----------------------|
+ | 0, 2, 4, 7, 7 | unspecified (padding) |
+
+ * Values array (`Struct<VarBinary, String>` element):
+ * Length: 7, Null count: 0
+ * Validity bitmap buffer: Not required
+
+ * Children arrays:
+ * field-0 array (`VarBinary` value)
+ * Length: 7, Null count: 6
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------|--------------------------|
+ | 0, 0, 0, 0, 1, 1, 1, 1 | unspecified (padding) |
+
+ * Values buffer (`00` -> variant null):
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-1 array (`String` typed_value)
+ * Length: 7, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 01110111 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------------|--------------------------|
+ | 0, 6, 11, 17, 17, 23, 28, 35 | unspecified (padding) |
+
+ * Values buffer:
+
+ | Bytes 0-35 | Bytes 36-63 |
+ |--------------------------------------|--------------------------|
+ | comedydramahorrorcomedydramaromance | unspecified (padding) |
+
+Shredding an Object
+'''''''''''''''''''
+
+Let's consider a simple JSON column of "events" which contain a field named ``event_type`` (a string)
Review Comment:
```suggestion
Let's consider a JSON column of "events" which contain a field named ``event_type`` (a string)
```
...after reading this, I may never be able to consider JSON simple again 🙂
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of: ::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field.
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like: ::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*. A null
+element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+The storage type to represent this in Arrow as a Variant extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: list<element: struct<
+ value: binary optional,
+ typed_value: string optional
+ > required> optional
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
+
+The data would then be stored in Arrow as follows: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000111 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary` Metadata)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary` Value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 0, 0, 1 | unspecified (padding) |
+
+ * Value buffer: (00 -> variant null)
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-2 array (`List<Struct<VarBinary, String>>` typed_value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00000111 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |-------------------|-----------------------|
+ | 0, 2, 4, 7, 7 | unspecified (padding) |
+
+ * Values array (`Struct<VarBinary, String>` element):
+ * Length: 7, Null count: 0
+ * Validity bitmap buffer: Not required
+
+ * Children arrays:
+ * field-0 array (`VarBinary` value)
+ * Length: 7, Null count: 6
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------|--------------------------|
+ | 0, 0, 0, 0, 1, 1, 1, 1 | unspecified (padding) |
+
+ * Values buffer (`00` -> variant null):
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-1 array (`String` typed_value)
+ * Length: 7, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 01110111 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------------|--------------------------|
+ | 0, 6, 11, 17, 17, 23, 28, 35 | unspecified (padding) |
+
+ * Values buffer:
+
+ | Bytes 0-35 | Bytes 36-63 |
+ |--------------------------------------|--------------------------|
+ | comedydramahorrorcomedydramaromance | unspecified (padding) |
+
+Shredding an Object
+'''''''''''''''''''
+
+Let's consider a simple JSON column of "events" which contain a field named ``event_type`` (a string)
+and a field named ``event_ts`` (a timestamp) that we wish to shred into separate columns, In Parquet,
+it could look something like this: ::
+
+ optional group event (VARIANT) {
+ required binary metadata;
+ optional binary value; # variant, remaining fields/values
+ optional group typed_value { # shredded fields for variant object
+ required group event_type { # event_type shredded field
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ required group event_ts { # event_ts shredded field
+ optional binary value;
+ optional int64 typed_value (TIMESTAMP(true, MICROS))
+ }
+ }
+ }
+
+We can then, fairly easily, translate this into the expected extension storage type: ::
Review Comment:
```suggestion
We can then translate this into the expected extension storage type: ::
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319986606
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _parquet_variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |-------------------------|--------------------------|
+ | 01 00 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
Review Comment:
Works for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319299331
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
Review Comment:
hypothetical. Its not really about parquet project defining extension types, its about anybody defining there own extension types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319560013
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
Review Comment:
Yes, there's a note below which states that they can be dictionary encoded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319586882
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
Review Comment:
Because the Parquet spec also explicitly states that the order isn't prescriptive and fields need to be accessed by name. In the interest of flexibility given that parquet, iceberg and spark implementations all state order doesn't matter and that fields are accessed by name, I followed that lead.
The other issue I can think of is that it would be a problem if a user creates a new parquet file from the resulting data after Arrow re-orders the columns, the Parquet schemas would no longer be equivalent/compatible. And even if we prescribe the order of the metadata/value/typed_value fields, you can't fix the order of the shredded fields. Simply put, given that this is defined in terms of an extension type based on struct arrays, it would be hard to enforce ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: add variant extension type docs [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2308896395
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
Review Comment:
```suggestion
* The storage type of this extension is a ``Struct`` that obeys the following rules:
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
Review Comment:
```suggestion
* A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
Review Comment:
```suggestion
.. note::
The fields may be in any order, and thus must be accessed by **name** not by *position*.
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
Review Comment:
```suggestion
Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of: ::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field.
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like: ::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*. A null
+element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+The storage type to represent this in Arrow as a Variant extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: list<element: struct<
+ value: binary optional,
+ typed_value: string optional
+ > required> optional
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
+
+The data would then be stored in Arrow as follows: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000111 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary` Metadata)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary` Value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 0, 0, 1 | unspecified (padding) |
+
+ * Value buffer: (00 -> variant null)
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-2 array (`List<Struct<VarBinary, String>>` typed_value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00000111 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |-------------------|-----------------------|
+ | 0, 2, 4, 7, 7 | unspecified (padding) |
+
+ * Values array (`Struct<VarBinary, String>` element):
+ * Length: 7, Null count: 0
+ * Validity bitmap buffer: Not required
+
+ * Children arrays:
+ * field-0 array (`VarBinary` value)
+ * Length: 7, Null count: 6
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------|--------------------------|
+ | 0, 0, 0, 0, 1, 1, 1, 1 | unspecified (padding) |
+
+ * Values buffer (`00` -> variant null):
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-1 array (`String` typed_value)
+ * Length: 7, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 01110111 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------------|--------------------------|
+ | 0, 6, 11, 17, 17, 23, 28, 35 | unspecified (padding) |
+
+ * Values buffer:
+
+ | Bytes 0-35 | Bytes 36-63 |
+ |--------------------------------------|--------------------------|
+ | comedydramahorrorcomedydramaromance | unspecified (padding) |
+
+Shredding an Object
+'''''''''''''''''''
+
+Let's consider a simple JSON column of "events" which contain a field named ``event_type`` (a string)
+and a field named ``event_ts`` (a timestamp) that we wish to shred into separate columns, In Parquet,
+it could look something like this: ::
+
+ optional group event (VARIANT) {
+ required binary metadata;
+ optional binary value; # variant, remaining fields/values
+ optional group typed_value { # shredded fields for variant object
+ required group event_type { # event_type shredded field
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ required group event_ts { # event_ts shredded field
+ optional binary value;
+ optional int64 typed_value (TIMESTAMP(true, MICROS))
+ }
+ }
+ }
+
+We can then, fairly easily, translate this into the expected extension storage type: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: struct<
+ event_type: struct<
+ value: binary optional,
+ typed_value: string optional
+ > required,
+ event_ts: struct<
+ value: binary optional,
+ typed_value: timestamp(us, UTC) optional
+ > required
+ > optional
+ >
+
+If a field *does not exist* in the variant object value, then both the **value** and **typed_value** columns for that row
+will be null. If a field is *present*, but the value is null, then **value** must contain a Variant null: *basic type*
+``0`` (primitive) and *physical type* ``0`` (null).
+
+It is *invalid* for both **value** and **typed_value** to be non-null for a given index. A reader can choose not to error
+in this scenario, but if so it **must** use the value in the **typed_value** column for that index.
+
+Let's consider the following series of objects: ::
+
+ {"event_type": "noop", "event_ts": 1729794114937}
+
+ {"event_type": "login", "event_ts": 1729794146402, "email": "user@example.com"}
+
+ {"error_msg": "malformed..."}
+
+ "malformed: not an object"
+
+ {"event_ts": 1729794240241, "click": "_button"}
+
+ {"event_ts": null, "event_ts": 1729794954163}
+
+ {"event_type": "noop", "event_ts": "2024-10-24"}
+
+ {}
+
+ null
+
+ *Entirely missing*
+
+To represent those values as a column of Variant values using the Variant extension type we get the following: ::
+
+ * Length: 10, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 11111111 | 00000001 | 0 (padding) |
+
+ * Children arrays
+ * field-0 array (`VarBinary` Metadata)
+ * Length: 10, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-43 (int32) | Bytes 44-63 |
+ |------------------------------------------|-------------------------|
+ | 0, 2, 11, 24, 26, 35, 37, 39, 41, 43, 45 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> version 1 empty metadata,
+ 01 01 00 XX ... -> Version 1, metadata with 1 elem, offset 0, offset XX == len(string), ... is dict string bytes)
+
+ | Bytes 0-1 | Bytes 2-10 | Bytes 11-23 | Bytes 24-25 | Bytes 26-34 |
+ |-------------------------------|-----------------------|-------------|-------------------|
+ | 01 00 | 01 01 00 05 email | 01 01 00 09 error_msg | 01 00 | 01 01 00 05 click |
+
+ | Bytes 35-36 | Bytes 37-38 | Bytes 39-40 | Bytes 41-42 | Bytes 43-44 | Bytes 45-63 |
+ |-------------|-------------|-------------|-------------|-------------|-----------------------|
+ | 01 00 | 01 00 | 01 00 | 01 00 | 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary` Value)
+ * Length: 10, Null count: 5
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |---------------------------|-----------|-----------------------|
+ | 00011110 | 00000001 | 0 (padding) |
+
+ * Offsets buffer (filled in based on lengths of encoded variants):
+
+ | ... |
+
+ * Value buffer:
+
+ | VariantEncode({"email": "user@email.com"}) | VariantEncode({"error_msg": "malformed..."}) |
+ | VariantEncode("malformed: not an object") | VariantEncode({"click": "_button"}) | 00 (null) |
+
+ * field-2 array (`Struct<...>` typed_value)
+ * Length: 10, Null count: 3
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 11110111 | 00000000 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`Struct<VarBinary, String>` event_type)
+ * Length: 10, Null count: 0
+ * Validity bitmap buffer: not required
+
+ * Children arrays
+ * field-0 array (`VarBinary` value)
+ * Length: 10, Null count: 9
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 01000000 | 00000000 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-43 (int32) | Bytes 44-63 |
+ |---------------------------------|-------------------------|
+ | 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1 | unspecified (padding) |
+
+ * Value buffer:
+
+ | Byte 0 | Bytes 1-63 |
+ |--------|------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-1 array (`String` typed_value)
+ * Length: 10, Null count: 7
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 01000011 | 00000000 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Byte 0-43 | Bytes 44-63 |
+ |-------------------------------------|------------------------|
+ | 0, 4, 9, 9, 9, 9, 9, 13, 13, 13, 13 | unspecified (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-3 | Bytes 4-8 | Bytes 9-12 | Bytes 13-63 |
+ |-----------|-----------|------------|------------------------|
+ | noop | login | noop | unspecified (padding) |
+
+
+ * field-1 array (`Struct<VarBinary, Timestamp>` event_ts)
+ * Length: 10, Null count: 0
+ * Validity bitmap buffer: not required
+
+ * Children arrays
+ * field-0 array (`VarBinary` value)
+ * Length: 10, Null count: 9
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 01000000 | 00000000 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-43 (int32) | Bytes 44-63 |
+ |---------------------------------|-------------------------|
+ | ... | unspecified (padding) |
+
+ * Value buffer:
+
+ | VariantEncode("2024-10-24") |
+
+ * field-1 array (`Timestamp(us, UTC)` typed_value)
+ * Length: 10, Null count: 6
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 00110011 | 00000000 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-7 | Bytes 8-15 | Bytes 16-31 | Bytes 32-39 | Bytes 40-47 | Bytes 48-63 |
+ |---------------|---------------|--------------|---------------|---------------|------------------------|
+ | 1729794114937 | 1729794146402 | unspecified | 1729794240241 | 1729794954163 | unspecified (padding) |
+
+
+Putting it all together
+'''''''''''''''''''''''
+
+As mentioned, the **typed_value** field associated with a Variant **value** can be of any shredded type. As a result,
+as long as we follow the original rules you can have an arbitrary number of nested levels based on how you want to
Review Comment:
```suggestion
as long as we follow the original rules we can have an arbitrary number of nested levels based on how you want to
```
nit but be consistent about 1st/2nd person?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
Review Comment:
```suggestion
* If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
Review Comment:
nit: should we be clear about labeling this "Parquet Variant"?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of: ::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field.
Review Comment:
We implicitly describe nulls here and then explicitly describe it below; would it make sense to explicitly introduce nulls first (and link to the docs) and then omit both of the other explanations?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of: ::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field.
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like: ::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*. A null
+element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+The storage type to represent this in Arrow as a Variant extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: list<element: struct<
+ value: binary optional,
+ typed_value: string optional
+ > required> optional
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
+
+The data would then be stored in Arrow as follows: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000111 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary` Metadata)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary` Value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 0, 0, 1 | unspecified (padding) |
+
+ * Value buffer: (00 -> variant null)
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-2 array (`List<Struct<VarBinary, String>>` typed_value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00000111 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |-------------------|-----------------------|
+ | 0, 2, 4, 7, 7 | unspecified (padding) |
+
+ * Values array (`Struct<VarBinary, String>` element):
+ * Length: 7, Null count: 0
+ * Validity bitmap buffer: Not required
+
+ * Children arrays:
+ * field-0 array (`VarBinary` value)
+ * Length: 7, Null count: 6
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------|--------------------------|
+ | 0, 0, 0, 0, 1, 1, 1, 1 | unspecified (padding) |
+
+ * Values buffer (`00` -> variant null):
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-1 array (`String` typed_value)
+ * Length: 7, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 01110111 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------------|--------------------------|
+ | 0, 6, 11, 17, 17, 23, 28, 35 | unspecified (padding) |
+
+ * Values buffer:
+
+ | Bytes 0-35 | Bytes 36-63 |
+ |--------------------------------------|--------------------------|
+ | comedydramahorrorcomedydramaromance | unspecified (padding) |
+
+Shredding an Object
+'''''''''''''''''''
+
+Let's consider a simple JSON column of "events" which contain a field named ``event_type`` (a string)
+and a field named ``event_ts`` (a timestamp) that we wish to shred into separate columns, In Parquet,
+it could look something like this: ::
Review Comment:
```suggestion
it could look something like this::
```
A double-colon [(1) leaves behind a colon (2) starts a code block, so no need to do both](https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html#literal-blocks)
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
Review Comment:
```suggestion
.. note::
It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2317613597
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
Review Comment:
```suggestion
.. _parquet_variant_extension:
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
Review Comment:
```suggestion
* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |-------------------------|--------------------------|
+ | 01 00 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field. A null
+ element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*.
+
+The storage type to represent this in Arrow as a Variant extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: list<element: struct<
+ value: binary nullable,
+ typed_value: string nullable
+ > required> nullable
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
+
+The data would then be stored in Arrow as follows::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000111 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary` metadata)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |-------------------------|--------------------------|
+ | 01 00 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary` value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 0, 0, 1 | unspecified (padding) |
+
+ * Value buffer: (00 -> variant null)
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-2 array (`List<Struct<VarBinary, String>>` typed_value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00000111 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |-------------------|-----------------------|
+ | 0, 2, 4, 7, 7 | unspecified (padding) |
+
+ * Values array (`Struct<VarBinary, String>` element):
+ * Length: 7, Null count: 0
+ * Validity bitmap buffer: Not required
+
+ * Children arrays:
+ * field-0 array (`VarBinary` value)
+ * Length: 7, Null count: 6
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------|--------------------------|
+ | 0, 0, 0, 0, 1, 1, 1, 1 | unspecified (padding) |
+
+ * Values buffer (`00` -> variant null):
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-1 array (`String` typed_value)
+ * Length: 7, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 01110111 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------------|--------------------------|
+ | 0, 6, 11, 17, 17, 23, 28, 35 | unspecified (padding) |
+
+ * Values buffer:
+
+ | Bytes 0-35 | Bytes 36-63 |
+ |--------------------------------------|--------------------------|
+ | comedydramahorrorcomedydramaromance | unspecified (padding) |
+
+Shredding an Object
+'''''''''''''''''''
+
+Let's consider a JSON column of "events" which contain a field named ``event_type`` (a string)
+and a field named ``event_ts`` (a timestamp) that we wish to shred into separate columns, In Parquet,
+it could look something like this::
+
+ optional group event (VARIANT) {
+ required binary metadata;
+ optional binary value; # variant, remaining fields/values
+ optional group typed_value { # shredded fields for variant object
+ required group event_type { # event_type shredded field
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ required group event_ts { # event_ts shredded field
+ optional binary value;
+ optional int64 typed_value (TIMESTAMP(true, MICROS))
+ }
+ }
+ }
+
+We can then translate this into the expected extension storage type::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: struct<
+ event_type: struct<
+ value: binary nullable,
+ typed_value: string nullable
+ > non-nullable,
+ event_ts: struct<
+ value: binary nullable,
+ typed_value: timestamp(us, UTC) nullable
+ > non-nullable
+ > nullable
+ >
+
+If a field *does not exist* in the variant object value, then both the **value** and **typed_value** columns for that row
+will be null. If a field is *present*, but the value is null, then **value** must contain a Variant null.
+
+It is *invalid* for both **value** and **typed_value** to be non-null for a given index. A reader can choose not to error
+in this scenario, but if so it **must** use the value in the **typed_value** column for that index.
+
+Let's consider the following series of objects::
+
+ {"event_type": "noop", "event_ts": 1729794114937}
+
+ {"event_type": "login", "event_ts": 1729794146402, "email": "user@example.com"}
+
+ {"error_msg": "malformed..."}
+
+ "malformed: not an object"
+
+ {"event_ts": 1729794240241, "click": "_button"}
+
+ {"event_ts": null, "event_ts": 1729794954163}
+
+ {"event_type": "noop", "event_ts": "2024-10-24"}
+
+ {}
+
+ null
+
+ *Entirely missing*
+
+To represent those values as a column of Variant values using the Variant extension type we get the following::
+
+ * Length: 10, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 11111111 | 00000001 | 0 (padding) |
+
+ * Children arrays
+ * field-0 array (`VarBinary` Metadata)
+ * Length: 10, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-43 (int32) | Bytes 44-63 |
+ |------------------------------------------|-------------------------|
+ | 0, 2, 11, 24, 26, 35, 37, 39, 41, 43, 45 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> version 1 empty metadata,
+ 01 01 00 XX ... -> Version 1, metadata with 1 elem, offset 0, offset XX == len(string), ... is dict string bytes)
+
+ | Bytes 0-1 | Bytes 2-10 | Bytes 11-23 | Bytes 24-25 | Bytes 26-34 |
+ |-------------------------------|-----------------------|-------------|-------------------|
+ | 01 00 | 01 01 00 05 email | 01 01 00 09 error_msg | 01 00 | 01 01 00 05 click |
+
+ | Bytes 35-36 | Bytes 37-38 | Bytes 39-40 | Bytes 41-42 | Bytes 43-44 | Bytes 45-63 |
+ |-------------|-------------|-------------|-------------|-------------|-----------------------|
+ | 01 00 | 01 00 | 01 00 | 01 00 | 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary` Value)
+ * Length: 10, Null count: 5
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |---------------------------|-----------|-----------------------|
+ | 00011110 | 00000001 | 0 (padding) |
+
+ * Offsets buffer (filled in based on lengths of encoded variants):
+
+ | ... |
+
+ * Value buffer:
+
+ | VariantEncode({"email": "user@email.com"}) | VariantEncode({"error_msg": "malformed..."}) |
+ | VariantEncode("malformed: not an object") | VariantEncode({"click": "_button"}) | 00 (null) |
+
+ * field-2 array (`Struct<...>` typed_value)
+ * Length: 10, Null count: 3
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 11110111 | 00000000 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`Struct<VarBinary, String>` event_type)
+ * Length: 10, Null count: 0
+ * Validity bitmap buffer: not required
+
+ * Children arrays
+ * field-0 array (`VarBinary` value)
+ * Length: 10, Null count: 9
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 01000000 | 00000000 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-43 (int32) | Bytes 44-63 |
+ |---------------------------------|-------------------------|
+ | 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1 | unspecified (padding) |
+
+ * Value buffer:
+
+ | Byte 0 | Bytes 1-63 |
+ |--------|------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-1 array (`String` typed_value)
+ * Length: 10, Null count: 7
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 01000011 | 00000000 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Byte 0-43 | Bytes 44-63 |
+ |-------------------------------------|------------------------|
+ | 0, 4, 9, 9, 9, 9, 9, 13, 13, 13, 13 | unspecified (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-3 | Bytes 4-8 | Bytes 9-12 | Bytes 13-63 |
+ |-----------|-----------|------------|------------------------|
+ | noop | login | noop | unspecified (padding) |
+
+
+ * field-1 array (`Struct<VarBinary, Timestamp>` event_ts)
+ * Length: 10, Null count: 0
+ * Validity bitmap buffer: not required
+
+ * Children arrays
+ * field-0 array (`VarBinary` value)
+ * Length: 10, Null count: 9
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 01000000 | 00000000 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-43 (int32) | Bytes 44-63 |
+ |---------------------------------|-------------------------|
+ | ... | unspecified (padding) |
+
+ * Value buffer:
+
+ | VariantEncode("2024-10-24") |
+
+ * field-1 array (`Timestamp(us, UTC)` typed_value)
+ * Length: 10, Null count: 6
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Byte 1 | Bytes 2-63 |
+ |--------------------------|-----------|-----------------------|
+ | 00110011 | 00000000 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-7 | Bytes 8-15 | Bytes 16-31 | Bytes 32-39 | Bytes 40-47 | Bytes 48-63 |
+ |---------------|---------------|--------------|---------------|---------------|------------------------|
+ | 1729794114937 | 1729794146402 | unspecified | 1729794240241 | 1729794954163 | unspecified (padding) |
+
+
+Putting it all together
+'''''''''''''''''''''''
+
+As mentioned, the **typed_value** field associated with a Variant **value** can be of any shredded type. As a result,
+as long as we follow the original rules we can have an arbitrary number of nested levels based on how you want to
+shred the object. For example, we might have a few more fields alongside **event_type** to shred out. Possibly an object
+that looks like this::
+
+ {
+ "event_type": "login",
+ “event_ts”: 1729794114937,
+ “location”: { “longitude”: 1.5, “latitude”: 5.5 },
+ “tags”: [“foo”, “bar”, “baz”]
Review Comment:
```suggestion
"event_ts": 1729794114937,
"location”: {"longitude": 1.5, "latitude": 5.5},
"tags": ["foo", "bar", "baz"]
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
Review Comment:
```suggestion
metadata: binary non-nullable,
value: binary nullable,
typed_value: int64 nullable
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |-------------------------|--------------------------|
+ | 01 00 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field. A null
+ element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*.
+
+The storage type to represent this in Arrow as a Variant extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: list<element: struct<
+ value: binary nullable,
+ typed_value: string nullable
+ > required> nullable
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
Review Comment:
```suggestion
As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
and **List** could also be **LargeList** or **ListView**.
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |-------------------------|--------------------------|
+ | 01 00 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field. A null
+ element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*.
+
+The storage type to represent this in Arrow as a Variant extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: list<element: struct<
+ value: binary nullable,
+ typed_value: string nullable
+ > required> nullable
Review Comment:
```suggestion
> non-nullable> nullable
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
Review Comment:
```suggestion
* Extension type parameters:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316398003
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
Review Comment:
We don't necessarily need the examples to be this long-winded. I mostly wanted to exemplify what the arrow arrays would look like in these specific scenarios. I can change this to limit the examples to just the Arrow schema definitions rather than the full descriptions of the array buffers if that would be better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316466118
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
Review Comment:
This should be spelled out clearly then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2318107124
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
Review Comment:
Probably because it mirrors what was decided on the Parquet side. However, there's a trade-off here:
1) if we are prescriptive on ordering, Parquet readers have to reorder fields when converting from Parquet columns if necessary, and applications can assume a given order
2) if we are not prescriptive on ordering, applications have to be order-agnostic
Option 1 sounds slightly better to me (and readers will have to be order-aware at some point if they grow higher-level variant access APIs anyway).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319710678
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
Review Comment:
I think run-end-encoding should be fine personally. Anyone object to updating this to say that REE is also permissible?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: add variant extension type docs [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2308402452
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +417,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it easy for systems to pass Variant data around without having to upgrade their Arrow version
Review Comment:
```suggestion
variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: add variant extension type docs [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3234761406
Higher up in this doc, can you please make this change:
> The specification text to be added *must* follow these requirements:
> 1. It *must* define a well-defined extension name which must start with an allowed prefix. The currently allowed prefixes are:
> - "``arrow.``" - For general-purpose canonical extension types.
> - "``parquet.``" - For canonical extension types that are intended primarily for interoperability with [Apache Parquet](https://parquet.apache.org) format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: add variant extension type docs [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3234762959
For uses of the word "variant" outside of code, please standardize the case on "Variant" or "variant" consistently. Right now it's a mix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2310316249
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of: ::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field.
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like: ::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*. A null
+element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+The storage type to represent this in Arrow as a Variant extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: list<element: struct<
+ value: binary optional,
+ typed_value: string optional
+ > required> optional
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
+
+The data would then be stored in Arrow as follows: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000111 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary` Metadata)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary` Value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 0, 0, 1 | unspecified (padding) |
+
+ * Value buffer: (00 -> variant null)
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-2 array (`List<Struct<VarBinary, String>>` typed_value)
+ * Length: 4, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00000111 | 0 (padding) |
+
+ * Offsets buffer (int32)
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |-------------------|-----------------------|
+ | 0, 2, 4, 7, 7 | unspecified (padding) |
+
+ * Values array (`Struct<VarBinary, String>` element):
+ * Length: 7, Null count: 0
+ * Validity bitmap buffer: Not required
+
+ * Children arrays:
+ * field-0 array (`VarBinary` value)
+ * Length: 7, Null count: 6
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 00001000 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------|--------------------------|
+ | 0, 0, 0, 0, 1, 1, 1, 1 | unspecified (padding) |
+
+ * Values buffer (`00` -> variant null):
+
+ | Bytes 0 | Bytes 1-63 |
+ |--------------------|--------------------------|
+ | 00 | unspecified (padding) |
+
+ * field-1 array (`String` typed_value)
+ * Length: 7, Null count: 1
+ * Validity bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|-------------|
+ | 01110111 | 0 (padding) |
+
+ * Offsets buffer (int32):
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------------------|--------------------------|
+ | 0, 6, 11, 17, 17, 23, 28, 35 | unspecified (padding) |
+
+ * Values buffer:
+
+ | Bytes 0-35 | Bytes 36-63 |
+ |--------------------------------------|--------------------------|
+ | comedydramahorrorcomedydramaromance | unspecified (padding) |
+
+Shredding an Object
+'''''''''''''''''''
+
+Let's consider a simple JSON column of "events" which contain a field named ``event_type`` (a string)
+and a field named ``event_ts`` (a timestamp) that we wish to shred into separate columns, In Parquet,
+it could look something like this: ::
Review Comment:
ah, i didn't realize it left behind the colon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2310353789
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``StructArray`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any *primitive type* or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary required, value: binary required>``
+* ``struct<value: binary optional, metadata: binary required>``
+* ``struct<metadata: dictionary<int8, binary> required, value: binary_view required>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be: ::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of: ::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as: ::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field.
Review Comment:
Is the updated version better?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,585 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Variant
+=======
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2312164211
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
Review Comment:
Is it case sensitive?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
Review Comment:
```suggestion
| Bytes 0-7 | Bytes 8-63 |
|-------------------------|--------------------------|
| 01 00 01 00 01 00 01 00 | unspecified (padding) |
```
Am I correct?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field. A null
+ element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*.
+
+The storage type to represent this in Arrow as a Variant extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: list<element: struct<
+ value: binary nullable,
+ typed_value: string nullable
+ > required> nullable
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
+
+The data would then be stored in Arrow as follows::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000111 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary` Metadata)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
Review Comment:
ditto
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field. A null
+ element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*.
+
+The storage type to represent this in Arrow as a Variant extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: list<element: struct<
+ value: binary nullable,
+ typed_value: string nullable
+ > required> nullable
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
+
+The data would then be stored in Arrow as follows::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000111 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary` Metadata)
Review Comment:
```suggestion
* field-0 array (`VarBinary` metadata)
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field. A null
+ element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*.
+
+The storage type to represent this in Arrow as a Variant extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: list<element: struct<
+ value: binary nullable,
+ typed_value: string nullable
+ > required> nullable
+ >
+
+.. note::
+
+ As usual, **Binary** could also be **LargeBinary** or **BinaryView**, **String** could also be **LargeString** or **StringView**,
+ and **List** could also be **LargeList** or **ListView**.
+
+The data would then be stored in Arrow as follows::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000111 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary` Metadata)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |--------------------|--------------------------|
+ | 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary` Value)
Review Comment:
```suggestion
* field-1 array (`VarBinary` value)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2314403344
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*.
Review Comment:
they should be case sensitive, yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3268474553
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2331266068
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,133 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _parquet_variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
Review Comment:
updated with this text. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3253712540
Sorry I have been out the last week -- I will review this shortly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319541766
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
Review Comment:
I'm late to the review party here but do we want to make sure we always refer to this specific Variant as "Parquet Variant" so we don't confuse it with a potential Arrow Variant?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
+ interoperability with `Apache Parquet <https://parquet.apache.org/>`__ format.
Review Comment:
```suggestion
interoperability with the `Apache Parquet <https://parquet.apache.org/>`__ format.
```
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _parquet_variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |-------------------------|--------------------------|
+ | 01 00 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
Review Comment:
This gets cut off in the rendered HTML, can we hard-wrap this or something?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,592 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _parquet_variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. ``INT``, ``STRING``)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: int64 nullable
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |-------------------------|--------------------------|
+ | 01 00 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> literal null, `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
+ |---------------------|--------------------------|
+ | 34, 00, 00, 100 | unspecified (padding) |
+
+.. note::
+
+ Notice that there is a variant ``literal null`` in the ``value`` array, this is due to the
+ `shredding specification <https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding>`__
+ so that a consumer can tell the difference between a *missing* field and a **null** field. A null
+ element must be encoded as a Variant null: *basic type* ``0`` (primitive) and *physical type* ``0`` (null).
+
+Shredding an Array
+''''''''''''''''''
+
+For our next example, we will represent a shredded array of strings. Let's consider a column that looks like: ::
+
+ ["comedy", "drama"], ["horror", null], ["comedy", "drama", "romance"], null
+
+Representing this shredded variant in Parquet could look like::
+
+ optional group tags (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional group typed_value (LIST) { # optional to allow null lists
+ repeated group list {
+ required group element { # shredded element
+ optional binary value;
+ optional binary typed_value (STRING);
+ }
+ }
+ }
+ }
+
+The array structure for Variant encoding does not allow missing elements, so all elements of the array must
+be *non-nullable*. As such, either **typed_value** or **value** (*but not both!*) must be *non-null*.
+
+The storage type to represent this in Arrow as a Variant extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: list<element: struct<
+ value: binary nullable,
+ typed_value: string nullable
+ > non-nullable> nullable
+ >
+
+.. note::
+
Review Comment:
Why bold?
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
Review Comment:
(Suggestion here would be replace standalone "Variant" with "Parquet Variant"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319876458
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
Review Comment:
Added a mapping table
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
Review Comment:
moved the examples to a separate document
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319565529
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
Review Comment:
The Arrow spec is also versioned, and the data in the metadata array is the raw Parquet Variant bytes which include the version number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319729955
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
Review Comment:
In a discussion about this in the Parquet sync meeting today, there was a general preference for `arrow.parquet.` because it does not expand the previously reserved namespaces and avoids possible name collisions. It sounds like we'll go with that unless there are well-reasoned objections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316299876
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
Review Comment:
The concrete example doesn't seem required to me? @wgtmac What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316321845
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
Review Comment:
Can it really be any primitive Arrow type, or should it have an equivalent type in the [Variant spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types)?
Perhaps something like:
> if the `typed_value` is a *primitive* type, it should correspond to one of the "Parquet equivalent types" mentioned in the [Variant encoding spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types)
or, even better, perhaps we want to have our own table mapping Arrow primitive types and Variant encoding types?
WDYT @tustvold @alamb @wgtmac ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2317512267
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
Review Comment:
if we go with nit above, I think `arrow.parquet` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2318109671
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
Review Comment:
I find it unlikely that the Parquet project would separately define its own Arrow extension types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3301294839
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6c0c3cd082b52660ae28b7bce6507a9f2d9f1091.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/50556464185) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3300458331
Thanks everyone for the reviews and assistance making this happen!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319724839
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
Review Comment:
> Option 1 sounds slightly better to me (and readers will have to reorder at some point if they grow higher-level variant access APIs anyway).
Readers don't need to perform any reordering for higher-level variant access APIs as those APIs wouldn't provide access by field position regardless and will just use maps by field names (which is what the current higher-level APIs in the Go implementation does)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2319877197
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
Review Comment:
moved the examples to a separate document
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
Review Comment:
Updated the wording
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316467118
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary required,
+ value: binary optional,
+ typed_value: int64 optional
+ >
+
+If we suppose a series of measurements consisting of::
Review Comment:
As I said above, my vote would be now on a separate document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2317516859
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,591 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows the rules outlined above (this allows for arbitrarily nested data).
+* Extension type parameters:
+
+ This type does not have any parameters.
+
+* Description of the serialization:
+
+ Extension metadata is an empty string.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with a preferred (*but not required*) index type of ``int8``.
+
+.. note::
+
+ The fields may be in any order, and thus must be accessed by **name** not by *position*. The field names are case sensitive.
+
+Examples
+--------
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
Review Comment:
can you remind me on why we decided not to be prescriptive here on ordering?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316398146
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * A field named ``typed_value`` which follows these same rules for the ``typed_value`` field.
+
+.. note::
+
+ It is also *permissible* for the ``metadata`` field to be dictionary-encoded with an index type of ``int8``.
Review Comment:
Ahem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316289807
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
Review Comment:
Er... so `typed_value` can be a List/LargeList/ListView of non-nullable Structs, or a Struct of Structs? This could be formulated more clearly.
Also, why a Struct of Structs instead of just a Struct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2316283784
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -417,7 +421,584 @@ better zero-copy compatibility with various systems that also store booleans usi
Metadata is an empty string.
-=========================
+.. _variant_extension:
+
+Parquet Variant
+===============
+
+Variant represents a value that may be one of:
+
+* Primitive: a type and corresponding value (e.g. INT, STRING)
+
+* Array: An ordered list of Variant values
+
+* Object: An unordered collection of string/Variant pairs (i.e. key/value pairs). An object may not contain duplicate keys
+
+Particularly, this provides a way to represent semi-structured data which is stored as a
+`Parquet Variant <https://github.com/apache/parquet-format/blob/master/VariantEncoding.md>`__ value within Arrow columns in
+a lossless fashion. This also provides the ability to represent `shredded <https://github.com/apache/parquet-format/blob/master/VariantShredding.md>`__
+variant values. This will make it possible for systems to pass Variant data around without having to upgrade their Arrow version
+or otherwise require special handling unless they want to directly interact with the encoded variant data. See the previous links
+to the Parquet format specification for details on what the actual binary values should look like.
+
+* Extension name: ``parquet.variant``.
+
+* The storage type of this extension is a ``Struct`` that obeys the following rules:
+
+ * A *non-nullable* field named ``metadata`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+
+ * At least one (or both) of the following:
+
+ * A field named ``value`` which is of type ``Binary``, ``LargeBinary``, or ``BinaryView``.
+ *(unshredded variants consist of just the ``metadata`` and ``value`` fields only)*
+
+ * A field named ``typed_value`` which can be any :term:`primitive type` or a ``List``, ``LargeList``, ``ListView`` or ``Struct``
+
+ * If the ``typed_value`` field is a *nested* type, its elements **must** be *non-nullable* and **must** be a ``Struct`` consisting of
+ at least one (or both) of the following:
Review Comment:
```suggestion
* If the ``typed_value`` field is a *nested* type, it **must** be a non-nullable ``Struct`` consisting of
at least one (or both) of the following:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "ianmcook (via GitHub)" <gi...@apache.org>.
ianmcook commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2317661819
##########
docs/source/format/CanonicalExtensions.rst:
##########
@@ -45,7 +45,11 @@ types:
* The specification text to be added *must* follow these requirements:
- 1) It *must* define a well-defined extension name starting with "``arrow.``".
+ 1) It *must* define a well-defined extension name starting with an allowed prefix.
+ The currently allowed prefixes are:
+ * "``arrow.``" - For general-purpose canonical extension types.
+ * "``parquet.``" - For canonical extension types that are intended primarily for
Review Comment:
A hypothetical conflict or a known real conflict?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3267333565
For anyone who missed it this proposal was voted and approved: https://mail.google.com/mail/u/0/#label/arrow-dev/FMfcgzQcpdfQqkZHshfWtmvwbctKTjGg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "codephage2020 (via GitHub)" <gi...@apache.org>.
codephage2020 commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2331934124
##########
docs/source/format/CanonicalExtensions/Examples.rst:
##########
@@ -0,0 +1,555 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_canonical_extension_examples:
+
+****************************
+Canonical Extension Examples
+****************************
+
+=========================
+Parquet Variant Extension
+=========================
+
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``arrow.parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: int64 nullable
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
+
+ | Bytes 0-7 | Bytes 8-63 |
+ |-------------------------|--------------------------|
+ | 01 00 01 00 01 00 01 00 | unspecified (padding) |
+
+ * field-1 array (`VarBinary`)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00000110 | 0 (padding) |
+
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 0, 1, 5, 5 | unspecified (padding) |
+
+ * Value buffer: (`00` -> null,
+ `0x13 0x6E 0x2F 0x61` -> variant encoding literal string "n/a")
+
+ | Bytes 0-4 | Bytes 5-63 |
+ |------------------------|--------------------------|
+ | 00 0x13 0x6E 0x2F 0x61 | unspecified (padding) |
+
+ * field-2 array (int64 array)
+ * Length: 4, Null count: 2
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001001 | 0 (padding) |
+
+ * Value buffer:
+
+ | Bytes 0-31 | Bytes 32-63 |
Review Comment:
A small suggestion to make tables look better
```suggestion
| Bytes 0-31 | Bytes 32-63 |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: [Docs][Format] Add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #47456:
URL: https://github.com/apache/arrow/pull/47456#discussion_r2334743821
##########
docs/source/format/CanonicalExtensions/Examples.rst:
##########
@@ -0,0 +1,555 @@
+.. Licensed to the Apache Software Foundation (ASF) under one
+.. or more contributor license agreements. See the NOTICE file
+.. distributed with this work for additional information
+.. regarding copyright ownership. The ASF licenses this file
+.. to you under the Apache License, Version 2.0 (the
+.. "License"); you may not use this file except in compliance
+.. with the License. You may obtain a copy of the License at
+
+.. http://www.apache.org/licenses/LICENSE-2.0
+
+.. Unless required by applicable law or agreed to in writing,
+.. software distributed under the License is distributed on an
+.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+.. KIND, either express or implied. See the License for the
+.. specific language governing permissions and limitations
+.. under the License.
+
+.. _format_canonical_extension_examples:
+
+****************************
+Canonical Extension Examples
+****************************
+
+=========================
+Parquet Variant Extension
+=========================
+
+
+Unshredded
+''''''''''
+
+The simplest case, an unshredded variant always consists of **exactly** two fields: ``metadata`` and ``value``. Any of
+the following storage types are valid (not an exhaustive list):
+
+* ``struct<metadata: binary non-nullable, value: binary nullable>``
+* ``struct<value: binary nullable, metadata: binary non-nullable>``
+* ``struct<metadata: dictionary<int8, binary> non-nullable, value: binary_view nullable>``
+
+Simple Shredding
+''''''''''''''''
+
+Suppose we have a Variant field named *measurement* and we want to shred the ``int64`` values into a separate column for efficiency.
+In Parquet, this could be represented as::
+
+ required group measurement (VARIANT) {
+ required binary metadata;
+ optional binary value;
+ optional int64 typed_value;
+ }
+
+Thus the corresponding storage type for the ``arrow.parquet.variant`` Arrow extension type would be::
+
+ struct<
+ metadata: binary non-nullable,
+ value: binary nullable,
+ typed_value: int64 nullable
+ >
+
+If we suppose a series of measurements consisting of::
+
+ 34, null, "n/a", 100
+
+The data should be stored/represented in Arrow as::
+
+ * Length: 4, Null count: 1
+ * Validity Bitmap buffer:
+
+ | Byte 0 (validity bitmap) | Bytes 1-63 |
+ |--------------------------|---------------|
+ | 00001011 | 0 (padding) |
+
+ * Children arrays:
+ * field-0 array (`VarBinary`)
+ * Length: 4, Null count: 0
+ * Offsets buffer:
+
+ | Bytes 0-19 | Bytes 20-63 |
+ |------------------|--------------------------|
+ | 0, 2, 4, 6, 8 | unspecified (padding) |
+
+ * Value buffer: (01 00 -> indicates version 1 empty metadata)
Review Comment:
> 01(byte[0]) indicate the header with offset_size_minus_one = 0, and 00(byte[1]) means the dictionary_size = 0, so there would no offset list and bytes following.
Correct
> There are four 01 00 in the meta, because we have four variant/row(34, null, "n/a", 100) in all here, am I understand right? If it's yes, does this mean we'll have ${lenght_of_row} pair of <header,dictionay_size, offset, bytes> in the metadata VarBinary
Correct. There should always be the same number of rows in the `metadata`, `value` and `typed_value` columns (if they exist)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46908: add variant extension type docs [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #47456:
URL: https://github.com/apache/arrow/pull/47456#issuecomment-3234732234
I'll wait for a few reviews on this before I send something on the mailing list to start a vote.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org