You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2024/05/16 12:05:09 UTC
[I] [Format][Docs] Clarify (remove?) usage of the term "logical types" [arrow]
jorisvandenbossche opened a new issue, #41691:
URL: https://github.com/apache/arrow/issues/41691
In several places in the Arrow specification and documentation we use the term "logical types", although we don't use it consistently and we don't actually have physical types (only physical layouts) to contrast it with.
### Current usage
The Columnar Format doc page has a section called "Logical Types" (https://github.com/apache/arrow/pull/41685) to contrast those types from the physical layouts:
> The [Schema.fbs](https://github.com/apache/arrow/blob/main/format/Schema.fbs) defines built-in logical types supported by the Arrow columnar format. Each logical type uses one of the above physical layouts. Nested logical types may have different physical layouts depending on the particular realization of the type.
It explains an Array as having a logical data type, where _"Each logical data type has a well-defined physical layout."_
The authoritative Schema.fbs also uses the term:
https://github.com/apache/arrow/blob/07a30d9a5784852187d100660325b8c12b4ff6c8/format/Schema.fbs#L18
although it uses the term also in a "correct" way (but incorrect in the way we define the term currently):
https://github.com/apache/arrow/blob/07a30d9a5784852187d100660325b8c12b4ff6c8/format/Schema.fbs#L101-L105
The Python docs (https://arrow.apache.org/docs/15.0/python/data.html#type-metadata):
> We use the name **logical type** because the **physical** storage may be the same for one or more types. For example, ``int64``, ``float64``, and ``timestamp[ms]`` all occupy 64 bits per value.
Further, in various implementations the term is obviously used as well.
In the Terminology section of the Columnar Format docs (https://arrow.apache.org/docs/15.0/format/Columnar.html#terminology), we define it as:
> **Logical type**: An application-facing semantic value type that is implemented using some physical layout. For example, Decimal values are stored as 16 bytes in a fixed-size binary layout. Similarly, strings can be stored as ``List<1-byte>``. A timestamp may be stored as 64-bit fixed-size layout.
which is mostly correct with our current usage ("using some physical layout"), but it is also confusing that it explains strings as ``List<1-byte>`` as we have a different physical layout used for strings
### Previous discussion
Generally we use the term relatively consistently to contrast "logical types" from the "physical layouts", but confusion around the terminology has come up regularly (what are "physical types" then? And extension types are essentially "logical types", but annotating our own logical types). This was specifically discussed in https://github.com/apache/arrow/issues/14752.
@amoeba proposed (https://github.com/apache/arrow/issues/14752#issuecomment-1550157549):
> Still some discussion to be had about avoiding "logical" vs. "physical" in favor of "types" and "layouts" and possibly updating the format docs comprehensively
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format][Docs] Clarify (remove?) usage of the term "logical types" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #41691:
URL: https://github.com/apache/arrow/issues/41691#issuecomment-2152840556
Issue resolved by pull request 41958
https://github.com/apache/arrow/pull/41958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format][Docs] Clarify (remove?) usage of the term "logical types" [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on issue #41691:
URL: https://github.com/apache/arrow/issues/41691#issuecomment-2115140776
I was definitely confused by the term "logical type" when I became involved with Arrow several years ago and came slowly to the understanding that every system seems to use the terms logical and physical in a slightly different way and that it is more of a spectrum than a dichotomy.
We also have the term "encoding" used in two different ways: "run end encoded" is a "type" but "dictionary" is basically a special case in Schema.fbs and in the Arrow C Data interface. Implementations typically present these both as "types" (e.g., `pa.dictionary()`).
I agree that it is confusing but I am not sure what it should be replaced with. I suppose they could just be called "types" and "layouts", perhaps with dictionary encoding being a layout rather than a type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format][Docs] Clarify (remove?) usage of the term "logical types" [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #41691: [Format][Docs] Clarify (remove?) usage of the term "logical types"
URL: https://github.com/apache/arrow/issues/41691
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Format][Docs] Clarify (remove?) usage of the term "logical types" [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on issue #41691:
URL: https://github.com/apache/arrow/issues/41691#issuecomment-2115427394
I would maybe generally talk about "data types" to not just have "types". For the layouts I think the adjective "physical" is still useful, or otherwise more consistently use "memory" as adjective.
So that would give "physical layouts" or "memory layouts" and "data types"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org