You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "westonpace (via GitHub)" <gi...@apache.org> on 2023/08/02 22:01:34 UTC
[GitHub] [arrow] westonpace opened a new pull request, #37000: GH-36999: [Format] Protoytpe language for alternative layouts (DO NOT MERGE)
westonpace opened a new pull request, #37000:
URL: https://github.com/apache/arrow/pull/37000
### Rationale for this change
See the GH issue and the ML discussion [here](https://lists.apache.org/thread/djl9dbd7qmozxtjpfzby40gg23x0o3wo).
### What changes are included in this PR?
I extend the format to describe alternative layouts
### Are these changes tested?
N/A
### Are there any user-facing changes?
In the specification, yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #37000: GH-36999: [Format] Protoytpe language for alternative layouts (DO NOT MERGE)
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #37000:
URL: https://github.com/apache/arrow/pull/37000#discussion_r1282467359
##########
docs/source/format/CDataInterface.rst:
##########
@@ -218,6 +218,16 @@ names and types of child fields are read from the child arrays.
| ``+us:I,J,...`` | sparse union with type ids I,J... | |
+------------------------+---------------------------------------------------+------------+
+Types that are encoded in an alternative layout have format strings ending with ``>``
+followed by a character describing the alternative layout. For example, an int32 column
+encoded in the stringy-view layout would have the format string ``I>~``.
Review Comment:
This approach could potentially be used for something like RLE as well where RLE is an official layout but can represent any type.
Another approach could be to treat alternative layouts as types and use something like `~:I` for the above similar to the way dense/sparse unions are handled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #37000: GH-36999: [Format] Protoytpe language for alternative layouts (DO NOT MERGE)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37000:
URL: https://github.com/apache/arrow/pull/37000#issuecomment-1663033519
:warning: GitHub issue #36999 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #37000: GH-36999: [Format] Protoytpe language for alternative layouts (DO NOT MERGE)
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #37000:
URL: https://github.com/apache/arrow/pull/37000#discussion_r1282467359
##########
docs/source/format/CDataInterface.rst:
##########
@@ -218,6 +218,16 @@ names and types of child fields are read from the child arrays.
| ``+us:I,J,...`` | sparse union with type ids I,J... | |
+------------------------+---------------------------------------------------+------------+
+Types that are encoded in an alternative layout have format strings ending with ``>``
+followed by a character describing the alternative layout. For example, an int32 column
+encoded in the stringy-view layout would have the format string ``I>~``.
Review Comment:
This approach could potentially be used for something like RLE as well where RLE is an official layout but can represent any type.
Another approach could be to treat alternative layouts as types and use something like `~:I` for the above similar to the way dense/sparse unions are handled. We wouldn't need a separate table in this case (`~` would just be added to one of the tables above)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-36999: [Format] Protoytpe language for alternative layouts (DO NOT MERGE) [arrow]
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace closed pull request #37000: GH-36999: [Format] Protoytpe language for alternative layouts (DO NOT MERGE)
URL: https://github.com/apache/arrow/pull/37000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #37000: GH-36999: [Format] Protoytpe language for alternative layouts (DO NOT MERGE)
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #37000:
URL: https://github.com/apache/arrow/pull/37000#discussion_r1282467359
##########
docs/source/format/CDataInterface.rst:
##########
@@ -218,6 +218,16 @@ names and types of child fields are read from the child arrays.
| ``+us:I,J,...`` | sparse union with type ids I,J... | |
+------------------------+---------------------------------------------------+------------+
+Types that are encoded in an alternative layout have format strings ending with ``>``
+followed by a character describing the alternative layout. For example, an int32 column
+encoded in the stringy-view layout would have the format string ``I>~``.
Review Comment:
This approach could potentially be used in places like RLE as well where data can
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org