You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/05 18:03:56 UTC

Re: [I] Investigate use of `DynamicFilters` in ballista [datafusion-ballista]

andygrove commented on issue #1375:
URL: https://github.com/apache/datafusion-ballista/issues/1375#issuecomment-4887070301

   A couple of concrete data points from SF100 TPC-H benchmarking that may help scope this:
   
   **Dynamic filter pushdown is currently explicitly disabled in Ballista.** `ballista/core/src/extension.rs:817` sets:
   
   ```rust
   .set_bool("datafusion.optimizer.enable_dynamic_filter_pushdown", false)
   ```
   
   So the first step here is likely understanding *why* it's off (presumably because a dynamic filter is computed at runtime on one node and can't cross a serialized stage boundary to reach the probe-side scan on another node) and what it would take to make it survive distributed planning/serialization — which lines up with the "integrate with shuffle read" goal above.
   
   **Why it's worth it — where Ballista currently loses.** In a SF100 comparison (2 executors × 8 slots), properly-parallelized Ballista (`target_partitions=32`, 552 s) roughly matches vanilla Spark (593 s) and trails Comet (340 s). Breaking it down per query, Ballista actually *wins* many queries but loses the **join/scan-heavy** ones — Q7 (~2.5×), Q14 (~2.3×), Q19 (~2.3×), Q20 (~2.0×). Those are precisely the queries a runtime/dynamic join filter (pushing the build-side hash/min-max down to the probe-side scan) targets, so this is likely one of the higher-ROI planner gaps for closing the Ballista→Comet distance.
   
   Happy to help benchmark before/after on those specific queries once there's something to try.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org