Re: [I] Make filter pushdown API more precise for their purpose [datafusion]

["jayshrivastava (via GitHub)" <gi...@apache.org>]

jayshrivastava commented on issue #18856:
URL: https://github.com/apache/datafusion/issues/18856#issuecomment-5359395043

   I'm interested in removing the execution-time check in `HashJoinExec` and `AggregateExec` to see if a filter was pushed down. I think this should be done at optimization time, not at execution time. This is the problem I'm running into in datafusion-distributed:
   ```
   worker 1
   HashJoinExec  (dynamic_filter_predicate)
       NetworkShuffleExec
   
   
   worker 2
   DataSourceExec (dynamic_filter_predicate)
   ```
   The hash join doesn't produce a filter because there's no filter used on that worker. However, I would expect that the join to produce dynamic filters because that decision should have been made during optimization time, before I split up the plan.
   
   I was wondering why this check even exists because `handle_child_pushdown_result` should handle it. It turns out pruning predicates return `No` indicating they weren't pushed down. 
   
   One alternative I considered was artificially pushing filters down to the network nodes by doing a plan rewrite:
   ```
   worker 1
   HashJoinExec  (dynamic_filter_predicate)
       NetworkShuffleExec  (dynamic_filter_predicate)
   
   
   worker 2
   DataSourceExec (dynamic_filter_predicate)
   ``` 
   This feels like a smell and I'm not sure if it works in other projects. I think other projects like ballista will run into this because they chop up plans as well. IMO it's worth adding a new state to remove runtime filter pushdown checks. I can try writing a small PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org