You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/09/07 14:46:25 UTC
[GitHub] [arrow-rs] tustvold opened a new issue, #2677: Arrow Row Format
tustvold opened a new issue, #2677:
URL: https://github.com/apache/arrow-rs/issues/2677
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
<!--
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
-->
I think this crate has pretty good stories for operating on individual columns, either by downcasting to a concrete type, or invoking a `dyn` kernel.
The stories for multi-column operations are substantially weaker, with patchy support for common multi-column operations such as sorts, groupings, aggregations, reassembly, etc... We have some pieces such as `MutableArrayData`, `DynComparator`, but they're not especially performant, making extensive use of dynamic dispatch at the row-level, nor easy to use.
**Describe the solution you'd like**
<!--
A clear and concise description of what you want to happen.
-->
Having a first-class row representation will not only allow us to implement more performant versions of existing kernels such as lexsort, but also provide a pretty compelling primitive to downstreams with which to implement more advanced operations such as streaming merges, joins, aggregates, etc... There is also precedent, with the C++ arrow library providing its own row format.
**Goals**
* Each row should be encoded as a single sequence of bytes
* Comparison of the byte arrays should be sufficient to establish ordering of the rows
* It should be possible to convert a selection of rows back to arrays
**Non-Goals**
* Support introspection or mutation of the row values
* Provide a stable encoding for FFI, IO, etc...
* Provide "optimal" encoding, rather a reasonable out-of-the-box baseline for common use-cases
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features you've considered.
-->
We could extend the row format in DataFusion, however, this would limit its benefits to DataFusion. I think a row-oriented representation is such a fundamental primitive that it makes sense for inclusion in arrow-rs, so that it can be both used in its kernels and by downstreams that don't make use of DataFusion.
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold closed issue #2677: Arrow Row Format
Posted by GitBox <gi...@apache.org>.
tustvold closed issue #2677: Arrow Row Format
URL: https://github.com/apache/arrow-rs/issues/2677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] v1gnesh commented on issue #2677: Arrow Row Format
Posted by GitBox <gi...@apache.org>.
v1gnesh commented on issue #2677:
URL: https://github.com/apache/arrow-rs/issues/2677#issuecomment-1242954235
Thanks for clarifying. As you say, I'll have to make a separate stream for each row-type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] v1gnesh commented on issue #2677: Arrow Row Format
Posted by GitBox <gi...@apache.org>.
v1gnesh commented on issue #2677:
URL: https://github.com/apache/arrow-rs/issues/2677#issuecomment-1242853511
Hello,
Would this help with processing row/record-based data, where the number of columns (that is, the structure of the row/record) vary depending on the type of row/record in the file. Example: An ndjson file where rows can be of `n` different variants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] tustvold commented on issue #2677: Arrow Row Format
Posted by GitBox <gi...@apache.org>.
tustvold commented on issue #2677:
URL: https://github.com/apache/arrow-rs/issues/2677#issuecomment-1242902278
This will not allow operations on data with multiple schema, same as with RecordBatch.
That being said, in the case of rows with different variants, nulls will be inserted by the JSON reader for the columns found in other variants but not present in the current record. The data is effectively unified to a single schema.
This trades off memory efficiency for the abilitybto efficiently process data in a columnar fashion, without per-value dynamic dispatch.
This is likely acceptable, however, partitioning the data so that different schema aren't interleaved will lead to better performance and memory efficiency. The row format could help with implementing this, but does not alter the nature of arrow schema
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org