[jira] [Commented] (ARROW-8199) [C++] Guidance for creating multi-column sort on Table example?

["Wes McKinney (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17090795#comment-17090795 ] 

Wes McKinney commented on ARROW-8199:
-------------------------------------

> Mainly I'd like to know if this looks like the direction you're thinking for arrow::DataFrame?

No, to be honest from a glance it's a different direction from what I've been thinking. My thoughts there actually are for the data frame internally to be a mix of yet-to-be-scanned Datasets (e.g. from CSV or Parquet files), manifest (materialized in-memory) chunked arrays, and unevaluated expressions. Analytics requests translate requests into physical query plans to be executed by the to-be-developed query engine. I haven't been able to give this my full attention since writing the design docs last year but I intend to spend a large fraction of my time on it the rest of the year.

The reasoning for wanting to push data frame operations into a query engine is to get around the memory use issues and performance problems associated with "eager evaluation" data frame libraries like pandas (for example, a join in pandas materializes the entire joined data frame in memory). There are similar issues around sorting (particular with the knowledge of what you want to do with the sorted data -- e.g. sort followed by a slice can be executed as a Top-K operation for substantially less memory use)

That said, I know a number of people have expressed interest in having STL interface layers in Arrow to the data structures. This would be a valuable thing to contribute to the project. It's not mutually exclusive with the stuff I wrote above but wanted to give some idea 

> [C++] Guidance for creating multi-column sort on Table example?
> ---------------------------------------------------------------
>
>                 Key: ARROW-8199
>                 URL: https://issues.apache.org/jira/browse/ARROW-8199
>             Project: Apache Arrow
>          Issue Type: Wish
>          Components: C++
>    Affects Versions: 0.16.0
>            Reporter: Scott Wilson
>            Priority: Minor
>              Labels: c++, newbie
>         Attachments: ArrowCsv.cpp
>
>
> I'm just coming up to speed with Arrow and am noticing a dearth of examples ... maybe I can help here.
> I'd like to implement multi-column sorting for Tables and just want to ensure that I'm not duplicating existing work or proposing a bad design.
> My thought was to create a Table-specific version of SortToIndices() where you can specify the columns and sort order.
> Then I'd create Array "views" that use the Indices to remap from the original Array values to the values in sorted order. (Original data is not sorted, but could be as a second step.) I noticed some of the array list variants keep offsets, but didn't see anything that supports remapping per a list of indices, but this may just be my oversight?
> Thanks in advance, Scott



--
This message was sent by Atlassian Jira
(v8.3.4#803005)