You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Jorge (Jira)" <ji...@apache.org> on 2020/07/12 18:05:00 UTC

[jira] [Commented] (ARROW-9420) Add repartition/shuffle plan

    [ https://issues.apache.org/jira/browse/ARROW-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156361#comment-17156361 ] 

Jorge commented on ARROW-9420:
------------------------------

I started some work in this direction on this branch, https://github.com/jorgecarleitao/arrow/tree/repartition, but I have not added the integration with SQL or logical plan, as I would like to know your opinions on this.

> Add repartition/shuffle plan
> ----------------------------
>
>                 Key: ARROW-9420
>                 URL: https://issues.apache.org/jira/browse/ARROW-9420
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Rust - DataFusion
>            Reporter: Jorge
>            Priority: Major
>
> Some operations (group by, join, over(window.partition_by)) greatly benefit from hash partitioning.
> This is a proposal to add hash partitioning (based on a expression) to this library, so that optimizers can be written to optimize the plan based on the required hashing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)