You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2024/06/04 14:09:27 UTC
[PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
pitrou opened a new pull request, #41958:
URL: https://github.com/apache/arrow/pull/41958
Also address GH-14752 by adding a table of data types with their respective parameters and the corresponding layouts.
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2147638593
:warning: GitHub issue #41691 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2147795595
Revision: f654194ceb8766e43f874f2555c8df8d0fa3625f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-c270be27f4](https://github.com/ursacomputing/crossbow/branches/all?query=actions-c270be27f4)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/9369717966/job/25794906677)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1629412427
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
+Arrow columnar format. Each data type uses a well-defined physical layout.
+
+`Schema.fbs`_ is the authoritative source for the description of the
+standard Arrow data types. However, we also provide the below table for
+convenience:
+
++--------------------+------------------------------+------------------------------------------------------------+
+| Type | Type Parameters *(1)* | Physical Memory Layout |
++====================+==============================+============================================================+
+| Null | | Null |
++--------------------+------------------------------+------------------------------------------------------------+
+| Boolean | | Fixed-size Primitive |
++--------------------+------------------------------+------------------------------------------------------------+
+| Int | * bit width | *" (same as above)* |
+| | * signedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Floating Point | * precision | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Decimal | * bit width | *"* |
+| | * scale | |
+| | * precision | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Date | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Time | * bit width *(2)* | *"* |
+| | * unit | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Timestamp | * unit | *"* |
+| | * timezone | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Interval | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Duration | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size Binary | * byte width | Fixed-size Binary |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary | | Variable-size Binary with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Binary | | Variable-size Binary with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary View | | Variable-size Binary View |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 View | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size List | * *value type* | Fixed-size List |
+| | * list size | |
++--------------------+------------------------------+------------------------------------------------------------+
+| List | * *value type* | Variable-size List with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List | * *value type* | Variable-size List with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| List View | * *value type* | Variable-size List View with 32-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List View | * *value type* | Variable-size List View with 64-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Struct | * *children* | Struct |
++--------------------+------------------------------+------------------------------------------------------------+
+| Map | * *children* | Variable-size List of Structs |
+| | * keys sortedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Union | * *children* | Dense or Sparse Union *(3)* |
+| | * mode | |
+| | * type ids | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Dictionary | * *index type* *(4)* | Dictionary Encoded |
+| | * *value type* | |
+| | * orderedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Run-End Encoded | * *run end type* *(5)* | Run-End Encoded |
+| | * *value type* | |
++--------------------+------------------------------+------------------------------------------------------------+
+
+* \(1) Type parameters listed in *italics* denote a data type's child types.
+
+* \(2) The *bit width* parameter of a Time type is technically redundant as
Review Comment:
this is nice to call out as I have noticed it before
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
Review Comment:
The `The` seems hanging to me
```suggestion
`Schema.fbs`_ defines built-in data types supported by the
```
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
+Arrow columnar format. Each data type uses a well-defined physical layout.
+
+`Schema.fbs`_ is the authoritative source for the description of the
+standard Arrow data types. However, we also provide the below table for
+convenience:
+
++--------------------+------------------------------+------------------------------------------------------------+
+| Type | Type Parameters *(1)* | Physical Memory Layout |
++====================+==============================+============================================================+
+| Null | | Null |
++--------------------+------------------------------+------------------------------------------------------------+
+| Boolean | | Fixed-size Primitive |
++--------------------+------------------------------+------------------------------------------------------------+
+| Int | * bit width | *" (same as above)* |
+| | * signedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Floating Point | * precision | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Decimal | * bit width | *"* |
+| | * scale | |
+| | * precision | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Date | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Time | * bit width *(2)* | *"* |
+| | * unit | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Timestamp | * unit | *"* |
+| | * timezone | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Interval | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Duration | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size Binary | * byte width | Fixed-size Binary |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary | | Variable-size Binary with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Binary | | Variable-size Binary with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary View | | Variable-size Binary View |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 View | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size List | * *value type* | Fixed-size List |
+| | * list size | |
++--------------------+------------------------------+------------------------------------------------------------+
+| List | * *value type* | Variable-size List with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List | * *value type* | Variable-size List with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| List View | * *value type* | Variable-size List View with 32-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List View | * *value type* | Variable-size List View with 64-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Struct | * *children* | Struct |
++--------------------+------------------------------+------------------------------------------------------------+
+| Map | * *children* | Variable-size List of Structs |
+| | * keys sortedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Union | * *children* | Dense or Sparse Union *(3)* |
+| | * mode | |
+| | * type ids | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Dictionary | * *index type* *(4)* | Dictionary Encoded |
+| | * *value type* | |
+| | * orderedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Run-End Encoded | * *run end type* *(5)* | Run-End Encoded |
+| | * *value type* | |
++--------------------+------------------------------+------------------------------------------------------------+
+
+* \(1) Type parameters listed in *italics* denote a data type's child types.
+
+* \(2) The *bit width* parameter of a Time type is technically redundant as
+ each *unit* mandates a single bit width.
+
+* \(3) Whether a Union type uses the Sparse or Dense layout is denoted by its
+ *mode* parameter.
+
+* \(4) The *index type* of a Dictionary type can only be a signed integer type
Review Comment:
As an aside, I think the Rust implementation permits any integer type as a dictionary key (not that I think we should change this document or the spec but maybe we could make arrow-rs simpler if we removed the unsigned key types from dictionaries)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2147881121
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2148096804
The HTML preview can be seen at http://crossbow.voltrondata.com/pr_docs/41958/format/Columnar.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1627814694
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
Review Comment:
Ahah, good question. In the context of the Flatbuffers serialization, yes. But there's only a small fixed number of possible parameters, so this is less obvious than for timestamps or nested types, where there's a virtual infinity of possibilities.
As @jorisvandenbossche rightfully pointed out below, the Flatbuffers serialization in the IPC format is only one possible representation of the type system; others could work differently; as a matter of fact, the [C Data Interface](https://arrow.apache.org/docs/format/CDataInterface.html#data-type-description-format-strings) uses entirely different format codes for these types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1626231574
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
+Arrow columnar format. Each data type uses a well-defined physical layout.
+
+`Schema.fbs`_ is the authoritative source for the description of the
+standard Arrow data types. However, we also provide the below table for
+convenience:
+
++--------------------+------------------------------+------------------------------------------------------------+
+| Type | Type Parameters *(1)* | Physical Memory Layout |
++====================+==============================+============================================================+
+| Null | | Null |
++--------------------+------------------------------+------------------------------------------------------------+
+| Boolean | | Fixed-size Primitive |
++--------------------+------------------------------+------------------------------------------------------------+
+| Int | * bit width | *" (same as above)* |
+| | * signedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Floating Point | * precision | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Decimal | * bit width | *"* |
+| | * scale | |
+| | * precision | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Date | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Time | * bit width *(2)* | *"* |
+| | * unit | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Timestamp | * unit | *"* |
+| | * timezone | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Interval | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Duration | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size Binary | * byte width | Fixed-size Binary |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary | | Variable-size Binary with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Binary | | Variable-size Binary with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary View | | Variable-size Binary View |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 View | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size List | * *value type* | Fixed-size List |
+| | * list size | |
++--------------------+------------------------------+------------------------------------------------------------+
+| List | * *value type* | Variable-size List with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List | * *value type* | Variable-size List with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| List View | * *value type* | Variable-size List View with 32-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List View | * *value type* | Variable-size List View with 64-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Struct | * *children* | Struct |
++--------------------+------------------------------+------------------------------------------------------------+
+| Map | * *children* | Variable-size List of Structs |
+| | * keys sortedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Union | * *children* | Dense or Sparse Union *(3)* |
+| | * mode | |
+| | * type ids | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Dictionary | * *index type* *(4)* | Dictionary Encoded |
+| | * *value type* | |
+| | * orderedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Run-End Encoded | * *run end type* *(5)* | Run-End Encoded |
+| | * *value type* | |
++--------------------+------------------------------+------------------------------------------------------------+
+
+* \(1) Type parameters listed in *italics* denote a data type's child types.
+
+* \(2) Whether a Union type uses the Sparse or Dense layout is denoted by its
+ *mode* parameter.
+
+* \(3) The *bit width* parameter of a Time type is technically redundant as
+ each *unit* mandates a single bit width.
+
+* \(4) The *index type* of a Dictionary type can only be a signed integer type
+ with width 8 to 64 bits.
+
+* \(5) The *run-end type* of a Dictionary type can only be a signed integer type
+ with width 16 to 64 bits.
+
+.. note::
+ Sometimes the term "logical type" is used to denote the Arrow data types
+ and distinguish them from their respective physical layouts. However,
+ unlike other type systems such as `Apache Parquet <https://parquet.apache.org/>`__'s,
+ the Arrow type system doesn't have separate notions of physical types and
+ logical types.
+
+ The Arrow type system separately provides
+ :ref:`exception types <format_metadata_extension_types>`, which allow
Review Comment:
```suggestion
:ref:`extention types <format_metadata_extension_types>`, which allow
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1626228048
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
+Arrow columnar format. Each data type uses a well-defined physical layout.
+
+`Schema.fbs`_ is the authoritative source for the description of the
+standard Arrow data types. However, we also provide the below table for
+convenience:
+
++--------------------+------------------------------+------------------------------------------------------------+
+| Type | Type Parameters *(1)* | Physical Memory Layout |
++====================+==============================+============================================================+
+| Null | | Null |
++--------------------+------------------------------+------------------------------------------------------------+
+| Boolean | | Fixed-size Primitive |
++--------------------+------------------------------+------------------------------------------------------------+
+| Int | * bit width | *" (same as above)* |
+| | * signedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Floating Point | * precision | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Decimal | * bit width | *"* |
+| | * scale | |
+| | * precision | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Date | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Time | * bit width *(2)* | *"* |
+| | * unit | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Timestamp | * unit | *"* |
+| | * timezone | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Interval | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Duration | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size Binary | * byte width | Fixed-size Binary |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary | | Variable-size Binary with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Binary | | Variable-size Binary with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary View | | Variable-size Binary View |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 View | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size List | * *value type* | Fixed-size List |
+| | * list size | |
++--------------------+------------------------------+------------------------------------------------------------+
+| List | * *value type* | Variable-size List with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List | * *value type* | Variable-size List with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| List View | * *value type* | Variable-size List View with 32-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List View | * *value type* | Variable-size List View with 64-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Struct | * *children* | Struct |
++--------------------+------------------------------+------------------------------------------------------------+
+| Map | * *children* | Variable-size List of Structs |
+| | * keys sortedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Union | * *children* | Dense or Sparse Union *(3)* |
+| | * mode | |
+| | * type ids | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Dictionary | * *index type* *(4)* | Dictionary Encoded |
+| | * *value type* | |
+| | * orderedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Run-End Encoded | * *run end type* *(5)* | Run-End Encoded |
+| | * *value type* | |
++--------------------+------------------------------+------------------------------------------------------------+
+
+* \(1) Type parameters listed in *italics* denote a data type's child types.
+
+* \(2) Whether a Union type uses the Sparse or Dense layout is denoted by its
+ *mode* parameter.
+
+* \(3) The *bit width* parameter of a Time type is technically redundant as
+ each *unit* mandates a single bit width.
Review Comment:
2 and 3 should be swapped according to the numbers used in the table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2160892633
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 93712bfc71a5013231b950b2b655d77b14f83fa7.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/26081176276) has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2147885888
Revision: 0f2b4587fd40b3d77b277c2157600e2465ed65f3
Submitted crossbow builds: [ursacomputing/crossbow @ actions-c9a11fc3b7](https://github.com/ursacomputing/crossbow/branches/all?query=actions-c9a11fc3b7)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/9370321736/job/25797002173)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2147790291
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2152640047
> I agree that it makes things a bit intimidating; also, reading a fbs file just to find prose documentation is not really a pleasant experience. Perhaps open a new issue for this?
Opened https://github.com/apache/arrow/issues/42011 about moving a lot of the prose content from Schema.fbs to Columnar.rst
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1626233178
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
+Arrow columnar format. Each data type uses a well-defined physical layout.
+
+`Schema.fbs`_ is the authoritative source for the description of the
+standard Arrow data types. However, we also provide the below table for
+convenience:
+
++--------------------+------------------------------+------------------------------------------------------------+
+| Type | Type Parameters *(1)* | Physical Memory Layout |
++====================+==============================+============================================================+
+| Null | | Null |
++--------------------+------------------------------+------------------------------------------------------------+
+| Boolean | | Fixed-size Primitive |
++--------------------+------------------------------+------------------------------------------------------------+
+| Int | * bit width | *" (same as above)* |
+| | * signedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Floating Point | * precision | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Decimal | * bit width | *"* |
+| | * scale | |
+| | * precision | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Date | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Time | * bit width *(2)* | *"* |
+| | * unit | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Timestamp | * unit | *"* |
+| | * timezone | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Interval | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Duration | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size Binary | * byte width | Fixed-size Binary |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary | | Variable-size Binary with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Binary | | Variable-size Binary with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary View | | Variable-size Binary View |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 View | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size List | * *value type* | Fixed-size List |
+| | * list size | |
++--------------------+------------------------------+------------------------------------------------------------+
+| List | * *value type* | Variable-size List with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List | * *value type* | Variable-size List with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| List View | * *value type* | Variable-size List View with 32-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List View | * *value type* | Variable-size List View with 64-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Struct | * *children* | Struct |
++--------------------+------------------------------+------------------------------------------------------------+
+| Map | * *children* | Variable-size List of Structs |
+| | * keys sortedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Union | * *children* | Dense or Sparse Union *(3)* |
+| | * mode | |
+| | * type ids | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Dictionary | * *index type* *(4)* | Dictionary Encoded |
+| | * *value type* | |
+| | * orderedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Run-End Encoded | * *run end type* *(5)* | Run-End Encoded |
+| | * *value type* | |
++--------------------+------------------------------+------------------------------------------------------------+
+
+* \(1) Type parameters listed in *italics* denote a data type's child types.
+
+* \(2) Whether a Union type uses the Sparse or Dense layout is denoted by its
+ *mode* parameter.
+
+* \(3) The *bit width* parameter of a Time type is technically redundant as
+ each *unit* mandates a single bit width.
+
+* \(4) The *index type* of a Dictionary type can only be a signed integer type
+ with width 8 to 64 bits.
+
+* \(5) The *run-end type* of a Dictionary type can only be a signed integer type
+ with width 16 to 64 bits.
+
+.. note::
+ Sometimes the term "logical type" is used to denote the Arrow data types
+ and distinguish them from their respective physical layouts. However,
+ unlike other type systems such as `Apache Parquet <https://parquet.apache.org/>`__'s,
+ the Arrow type system doesn't have separate notions of physical types and
+ logical types.
+
+ The Arrow type system separately provides
+ :ref:`exception types <format_metadata_extension_types>`, which allow
+ adorning standard Arrow data types with richer application-facing semantics
Review Comment:
"annotating" instead of "adorning" ? (I hadn't yet heard of adorning in this context, but yeah, not a native speaker ;))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1626230449
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
+Arrow columnar format. Each data type uses a well-defined physical layout.
+
+`Schema.fbs`_ is the authoritative source for the description of the
+standard Arrow data types. However, we also provide the below table for
+convenience:
+
++--------------------+------------------------------+------------------------------------------------------------+
+| Type | Type Parameters *(1)* | Physical Memory Layout |
++====================+==============================+============================================================+
+| Null | | Null |
++--------------------+------------------------------+------------------------------------------------------------+
+| Boolean | | Fixed-size Primitive |
++--------------------+------------------------------+------------------------------------------------------------+
+| Int | * bit width | *" (same as above)* |
+| | * signedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Floating Point | * precision | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Decimal | * bit width | *"* |
+| | * scale | |
+| | * precision | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Date | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Time | * bit width *(2)* | *"* |
+| | * unit | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Timestamp | * unit | *"* |
+| | * timezone | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Interval | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Duration | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size Binary | * byte width | Fixed-size Binary |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary | | Variable-size Binary with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Binary | | Variable-size Binary with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary View | | Variable-size Binary View |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 View | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size List | * *value type* | Fixed-size List |
+| | * list size | |
++--------------------+------------------------------+------------------------------------------------------------+
+| List | * *value type* | Variable-size List with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List | * *value type* | Variable-size List with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| List View | * *value type* | Variable-size List View with 32-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List View | * *value type* | Variable-size List View with 64-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Struct | * *children* | Struct |
++--------------------+------------------------------+------------------------------------------------------------+
+| Map | * *children* | Variable-size List of Structs |
+| | * keys sortedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Union | * *children* | Dense or Sparse Union *(3)* |
+| | * mode | |
+| | * type ids | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Dictionary | * *index type* *(4)* | Dictionary Encoded |
+| | * *value type* | |
+| | * orderedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Run-End Encoded | * *run end type* *(5)* | Run-End Encoded |
+| | * *value type* | |
++--------------------+------------------------------+------------------------------------------------------------+
+
+* \(1) Type parameters listed in *italics* denote a data type's child types.
+
+* \(2) Whether a Union type uses the Sparse or Dense layout is denoted by its
+ *mode* parameter.
+
+* \(3) The *bit width* parameter of a Time type is technically redundant as
+ each *unit* mandates a single bit width.
+
+* \(4) The *index type* of a Dictionary type can only be a signed integer type
+ with width 8 to 64 bits.
+
+* \(5) The *run-end type* of a Dictionary type can only be a signed integer type
Review Comment:
```suggestion
* \(5) The *run-end type* of a Run End Encoded type can only be a signed integer type
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2147686848
Also cc @alamb @tustvold @paleolimbot @wesm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1626259266
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
+Arrow columnar format. Each data type uses a well-defined physical layout.
+
+`Schema.fbs`_ is the authoritative source for the description of the
+standard Arrow data types. However, we also provide the below table for
+convenience:
+
++--------------------+------------------------------+------------------------------------------------------------+
+| Type | Type Parameters *(1)* | Physical Memory Layout |
++====================+==============================+============================================================+
+| Null | | Null |
++--------------------+------------------------------+------------------------------------------------------------+
+| Boolean | | Fixed-size Primitive |
++--------------------+------------------------------+------------------------------------------------------------+
+| Int | * bit width | *" (same as above)* |
+| | * signedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Floating Point | * precision | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Decimal | * bit width | *"* |
+| | * scale | |
+| | * precision | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Date | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Time | * bit width *(2)* | *"* |
+| | * unit | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Timestamp | * unit | *"* |
+| | * timezone | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Interval | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Duration | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size Binary | * byte width | Fixed-size Binary |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary | | Variable-size Binary with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Binary | | Variable-size Binary with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary View | | Variable-size Binary View |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 View | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size List | * *value type* | Fixed-size List |
+| | * list size | |
++--------------------+------------------------------+------------------------------------------------------------+
+| List | * *value type* | Variable-size List with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List | * *value type* | Variable-size List with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| List View | * *value type* | Variable-size List View with 32-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List View | * *value type* | Variable-size List View with 64-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Struct | * *children* | Struct |
++--------------------+------------------------------+------------------------------------------------------------+
+| Map | * *children* | Variable-size List of Structs |
+| | * keys sortedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Union | * *children* | Dense or Sparse Union *(3)* |
+| | * mode | |
+| | * type ids | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Dictionary | * *index type* *(4)* | Dictionary Encoded |
+| | * *value type* | |
+| | * orderedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Run-End Encoded | * *run end type* *(5)* | Run-End Encoded |
+| | * *value type* | |
++--------------------+------------------------------+------------------------------------------------------------+
+
+* \(1) Type parameters listed in *italics* denote a data type's child types.
+
+* \(2) Whether a Union type uses the Sparse or Dense layout is denoted by its
+ *mode* parameter.
+
+* \(3) The *bit width* parameter of a Time type is technically redundant as
+ each *unit* mandates a single bit width.
+
+* \(4) The *index type* of a Dictionary type can only be a signed integer type
+ with width 8 to 64 bits.
+
+* \(5) The *run-end type* of a Dictionary type can only be a signed integer type
+ with width 16 to 64 bits.
+
+.. note::
+ Sometimes the term "logical type" is used to denote the Arrow data types
+ and distinguish them from their respective physical layouts. However,
+ unlike other type systems such as `Apache Parquet <https://parquet.apache.org/>`__'s,
+ the Arrow type system doesn't have separate notions of physical types and
+ logical types.
+
+ The Arrow type system separately provides
+ :ref:`exception types <format_metadata_extension_types>`, which allow
+ adorning standard Arrow data types with richer application-facing semantics
Review Comment:
Yes, "annotating" is probably clearer :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1629607303
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
+Arrow columnar format. Each data type uses a well-defined physical layout.
+
+`Schema.fbs`_ is the authoritative source for the description of the
+standard Arrow data types. However, we also provide the below table for
+convenience:
+
++--------------------+------------------------------+------------------------------------------------------------+
+| Type | Type Parameters *(1)* | Physical Memory Layout |
++====================+==============================+============================================================+
+| Null | | Null |
++--------------------+------------------------------+------------------------------------------------------------+
+| Boolean | | Fixed-size Primitive |
++--------------------+------------------------------+------------------------------------------------------------+
+| Int | * bit width | *" (same as above)* |
+| | * signedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Floating Point | * precision | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Decimal | * bit width | *"* |
+| | * scale | |
+| | * precision | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Date | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Time | * bit width *(2)* | *"* |
+| | * unit | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Timestamp | * unit | *"* |
+| | * timezone | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Interval | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Duration | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size Binary | * byte width | Fixed-size Binary |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary | | Variable-size Binary with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Binary | | Variable-size Binary with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary View | | Variable-size Binary View |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 View | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size List | * *value type* | Fixed-size List |
+| | * list size | |
++--------------------+------------------------------+------------------------------------------------------------+
+| List | * *value type* | Variable-size List with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List | * *value type* | Variable-size List with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| List View | * *value type* | Variable-size List View with 32-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List View | * *value type* | Variable-size List View with 64-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Struct | * *children* | Struct |
++--------------------+------------------------------+------------------------------------------------------------+
+| Map | * *children* | Variable-size List of Structs |
+| | * keys sortedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Union | * *children* | Dense or Sparse Union *(3)* |
+| | * mode | |
+| | * type ids | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Dictionary | * *index type* *(4)* | Dictionary Encoded |
+| | * *value type* | |
+| | * orderedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Run-End Encoded | * *run end type* *(5)* | Run-End Encoded |
+| | * *value type* | |
++--------------------+------------------------------+------------------------------------------------------------+
+
+* \(1) Type parameters listed in *italics* denote a data type's child types.
+
+* \(2) The *bit width* parameter of a Time type is technically redundant as
+ each *unit* mandates a single bit width.
+
+* \(3) Whether a Union type uses the Sparse or Dense layout is denoted by its
+ *mode* parameter.
+
+* \(4) The *index type* of a Dictionary type can only be a signed integer type
Review Comment:
Now that I re-read `Schema.fbs`, I realize this wording is overly strict. Here is what that file says:
> To maximize cross-language compatibility and performance, implementations are recommended to prefer signed integer types over unsigned integer types and to avoid uint64 indices unless they are required by an application.
I'll change to a more lax wording.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #41958:
URL: https://github.com/apache/arrow/pull/41958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2150005914
> I agree that it makes things a bit intimidating; also, reading a fbs file just to find prose documentation is not really a pleasant experience.
I agree and am very glad that this documentation will now be in a place where people are likely to actually see it 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1629607803
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
Review Comment:
Good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2147685578
I reused @AlenkaF 's idea of a table here, but I changed the contents quite a bit.
Some open questions:
* for the type parameters, should we use a description in natural language (such as "bit width" or "signedness") or the physical parameter names in `Schema.fbs` (such as `bitWidth` or `is_signed`)?
* should we explain the type parameters in full (this would result in a more lengthy though more informative doc), or should we just keep a couple explanatory notes as here?
* should we repeat the term "Layout" under the "Physical Memory Layout" column? For example, write "Fixed-size Primitive Layout" instead of the terser "Fixed-size Primitive".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1627810814
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
Review Comment:
Would integers or floating point types count here? (i.e., types that implementations have chosen not to parameterize for convenience but that are "parameterized" in this document?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1626343608
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1627814694
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
Review Comment:
Ahah, good question. In the context of the Flatbuffers serialization, yes. But there's only a small fixed number of possible parameters, so this is less obvious than for timestamps or nested types, where there's a virtual infinity of possibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2147676130
cc @jorisvandenbossche @AlenkaF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2149923401
> * should we explain the type parameters in full (this would result in a more lengthy though more informative doc), or should we just keep a couple explanatory notes as here?
I think for the content you created here (i.e. the overview table), the current level of detail is fine.
For me it is a separate question if we want to keep Schema.fbs as the "authoritative source" for the explanation of those parameters. Personally, I don't like that this is "hidden" in a fbs file which is actually specific for IPC (there are a few things in that file that are not generic for the columnar format). I think ideally we would move the explanation of the parameters in the fbs file to the actual format specification, but then the question is how to deal with the duplication with the fbs file (we don't want two places to keep in sync, but would it be fine to cut down the content in the fbs file largely?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #41958:
URL: https://github.com/apache/arrow/pull/41958#discussion_r1626259815
##########
docs/source/format/Columnar.rst:
##########
@@ -70,21 +70,131 @@ concepts, here is a small glossary to help disambiguate.
without taking into account any value semantics. For example, a
32-bit signed integer array and 32-bit floating point array have the
same layout.
-* **Parent** and **child arrays**: names to express relationships
- between physical value arrays in a nested type structure. For
- example, a ``List<T>``-type parent array has a T-type array as its
- child (see more on lists below).
+* **Data type**: An application-facing semantic value type that is
+ implemented using some physical layout. For example, Decimal128
+ values are stored as 16 bytes in a fixed-size binary
+ layout. A timestamp may be stored as 64-bit fixed-size layout.
* **Primitive type**: a data type having no child types. This includes
such types as fixed bit-width, variable-size binary, and null types.
* **Nested type**: a data type whose full structure depends on one or
more other child types. Two fully-specified nested types are equal
if and only if their child types are equal. For example, ``List<U>``
is distinct from ``List<V>`` iff U and V are different types.
-* **Logical type**: An application-facing semantic value type that is
- implemented using some physical layout. For example, Decimal
- values are stored as 16 bytes in a fixed-size binary
- layout. Similarly, strings can be stored as ``List<1-byte>``. A
- timestamp may be stored as 64-bit fixed-size layout.
+* **Parent** and **child arrays**: names to express relationships
+ between physical value arrays in a nested type structure. For
+ example, a ``List<T>``-type parent array has a T-type array as its
+ child (see more on lists below).
+* **Parametric type**: a type which requires additional parameters
+ for full determination of its semantics. For example, all nested types
+ are parametric by construction. A timestamp is also parametric as it needs
+ a unit (such as microseconds) and a timezone.
+
+Data Types
+==========
+
+The `Schema.fbs`_ defines built-in data types supported by the
+Arrow columnar format. Each data type uses a well-defined physical layout.
+
+`Schema.fbs`_ is the authoritative source for the description of the
+standard Arrow data types. However, we also provide the below table for
+convenience:
+
++--------------------+------------------------------+------------------------------------------------------------+
+| Type | Type Parameters *(1)* | Physical Memory Layout |
++====================+==============================+============================================================+
+| Null | | Null |
++--------------------+------------------------------+------------------------------------------------------------+
+| Boolean | | Fixed-size Primitive |
++--------------------+------------------------------+------------------------------------------------------------+
+| Int | * bit width | *" (same as above)* |
+| | * signedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Floating Point | * precision | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Decimal | * bit width | *"* |
+| | * scale | |
+| | * precision | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Date | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Time | * bit width *(2)* | *"* |
+| | * unit | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Timestamp | * unit | *"* |
+| | * timezone | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Interval | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Duration | * unit | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size Binary | * byte width | Fixed-size Binary |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary | | Variable-size Binary with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Binary | | Variable-size Binary with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large Utf8 | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Binary View | | Variable-size Binary View |
++--------------------+------------------------------+------------------------------------------------------------+
+| Utf8 View | | *"* |
++--------------------+------------------------------+------------------------------------------------------------+
+| Fixed-Size List | * *value type* | Fixed-size List |
+| | * list size | |
++--------------------+------------------------------+------------------------------------------------------------+
+| List | * *value type* | Variable-size List with 32-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List | * *value type* | Variable-size List with 64-bit offsets |
++--------------------+------------------------------+------------------------------------------------------------+
+| List View | * *value type* | Variable-size List View with 32-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Large List View | * *value type* | Variable-size List View with 64-bit offsets and sizes |
++--------------------+------------------------------+------------------------------------------------------------+
+| Struct | * *children* | Struct |
++--------------------+------------------------------+------------------------------------------------------------+
+| Map | * *children* | Variable-size List of Structs |
+| | * keys sortedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Union | * *children* | Dense or Sparse Union *(3)* |
+| | * mode | |
+| | * type ids | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Dictionary | * *index type* *(4)* | Dictionary Encoded |
+| | * *value type* | |
+| | * orderedness | |
++--------------------+------------------------------+------------------------------------------------------------+
+| Run-End Encoded | * *run end type* *(5)* | Run-End Encoded |
+| | * *value type* | |
++--------------------+------------------------------+------------------------------------------------------------+
+
+* \(1) Type parameters listed in *italics* denote a data type's child types.
+
+* \(2) Whether a Union type uses the Sparse or Dense layout is denoted by its
+ *mode* parameter.
+
+* \(3) The *bit width* parameter of a Time type is technically redundant as
+ each *unit* mandates a single bit width.
+
+* \(4) The *index type* of a Dictionary type can only be a signed integer type
+ with width 8 to 64 bits.
+
+* \(5) The *run-end type* of a Dictionary type can only be a signed integer type
+ with width 16 to 64 bits.
+
+.. note::
+ Sometimes the term "logical type" is used to denote the Arrow data types
+ and distinguish them from their respective physical layouts. However,
+ unlike other type systems such as `Apache Parquet <https://parquet.apache.org/>`__'s,
+ the Arrow type system doesn't have separate notions of physical types and
+ logical types.
+
+ The Arrow type system separately provides
+ :ref:`exception types <format_metadata_extension_types>`, which allow
Review Comment:
"extension" actually :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-41691: [Doc] Remove notion of "logical type" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #41958:
URL: https://github.com/apache/arrow/pull/41958#issuecomment-2149956326
> For me it is a separate question if we want to keep Schema.fbs as the "authoritative source" for the explanation of those parameters. Personally, I don't like that this is "hidden" in a fbs file which is actually specific for IPC (there are a few things in that file that are not generic for the columnar format).
I agree that it makes things a bit intimidating; also, reading a fbs file just to find prose documentation is not really a pleasant experience. Perhaps open a new issue for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org