[jira] [Resolved] (ARROW-10547) [Rust][DataFusion] Filter pushdown loses filters if below a user defined node

["Jorge Leitão (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jorge Leitão resolved ARROW-10547.
----------------------------------
    Fix Version/s: 3.0.0
       Resolution: Fixed

Issue resolved by pull request 8633
[https://github.com/apache/arrow/pull/8633]

> [Rust][DataFusion] Filter pushdown loses filters if below a user defined node
> -----------------------------------------------------------------------------
>
>                 Key: ARROW-10547
>                 URL: https://issues.apache.org/jira/browse/ARROW-10547
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Rust - DataFusion
>            Reporter: Andrew Lamb
>            Assignee: Andrew Lamb
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.0.0
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> I have a LogicalPlan like this (where "Extension" is some extension node type):
> {code}
>     Extension [non_null_column:Utf8]
>       Filter: #state Eq Utf8("MA") [city:Utf8;N, state:Utf8;N, borough:Utf8;N]
>         InMemoryScan: projection=None [city:Utf8;N, state:Utf8;N, borough:Utf8;N]
> {code}
> When I run this plan through {{ExecutionContext::optimize}} the plan that results is as follows (note the filter has been lost):
> {code}
>     Extension [non_null_column:Utf8]
>       InMemoryScan: projection=Some([0, 1, 2]) [city:Utf8;N, state:Utf8;N, borough:Utf8;N]
> {code}
> I have debugged the problem and the root cause of the issue is that the FilterPushdown logic is not recursing into the inputs of user defined nodes. I will get a PR up showing and fixing the problem.
> More generally, I am not sure how / if the filter pushdown logic would work if there are multiple child inputs as no existing LogicalPlan variant his multiple inputs at this time. We will have to handle this when / if we want to support joins (typically filters get  pushed down each join input if possible)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)