[jira] [Created] (ARROW-9732) [Rust] [DataFusion] Add "Physical Planner" type thing which can do optimizations

["Andrew Lamb (Jira)" <ji...@apache.org>]

Andrew Lamb created ARROW-9732:
----------------------------------

             Summary: [Rust] [DataFusion] Add "Physical Planner" type thing which can do optimizations
                 Key: ARROW-9732
                 URL: https://issues.apache.org/jira/browse/ARROW-9732
             Project: Apache Arrow
          Issue Type: Improvement
            Reporter: Andrew Lamb


[~andygrove] implemented what I would describe as a "limit pushdown" optimization within Limit here: https://github.com/apache/arrow/pull/7958#discussion_r470175966

However, it was implemented by directly instantiating Partition objects during plan execution. This "pick the top N from each partition and then pick the top N from the merged result" is an example of operator pushdown that could be done at planning time

This ticket tracks the work to add some way to represent the in the planning stage, rather than execution, in order to open up more optimization opportunities. 

One example of pushdown that could potentially happen at planning time would be pushing the limit down past Projections for example. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)