You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Tamar-Posen (via GitHub)" <gi...@apache.org> on 2026/01/12 12:29:10 UTC
[I] Support Runtime Pruning using Dynamic Filters and Inverted Index [datafusion]
Tamar-Posen opened a new issue, #19764:
URL: https://github.com/apache/datafusion/issues/19764
### Is your feature request related to a problem or challenge?
Using DataFusion 51 with HashJoin dynamic filters enabled, the performance gains are significant.
I’d like to extend this mechanism to leverage an inverted index, currently implemented in a custom TableProvider, for runtime pruning.
Architectural conflict:
Standard pruning occurs during logical/physical planning, but the values required for inverted index lookups (join keys - dynamic filter) are available only during the execution phase, after the build side of the hash join completes.
### Describe the solution you'd like
Introduce a custom execution node (e.g. IndexPruningExec) that wraps the DataSourceExec on the probe side of the join:
- Wait for dynamic filter values produced by HashJoinExec
- Query the inverted index using those values
- Dynamically prune files/row groups before executing the underlying DataSourceExec.
Questions
- Does this “wrapper exec” approach align with DataFusion’s long-term execution model?
- Is there an existing extension point intended for this kind of late-binding runtime pruning?
- Are there alternative designs (e.g., a pruning or predicate hook) that would be a better fit?
### Describe alternatives you've considered
Physical Optimizer Rules: Not viable because they run before dynamic filter values are produced.
Logical Sideways Information Passing: Too complex, duplicates work already done by the join's hash table, etc
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support Runtime Pruning using Dynamic Filters and Inverted Index [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on issue #19764:
URL: https://github.com/apache/datafusion/issues/19764#issuecomment-3741473735
I don’t know the answer to the architectural questions yet, but I’m also working on a pruning-related project (https://github.com/apache/datafusion/issues/19487). I’m curious about the motivation for using an inverted index here: besides it can be faster when pruning a large number of containers, are there other benefits or use cases you have in mind? Could you share more context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support Runtime Pruning using Dynamic Filters and Inverted Index [datafusion]
Posted by "Tamar-Posen (via GitHub)" <gi...@apache.org>.
Tamar-Posen commented on issue #19764:
URL: https://github.com/apache/datafusion/issues/19764#issuecomment-3742533363
Yes, it's exactly about scale and metadata overhead.
We operate on 100TB+ datasets with millions of Parquet files, where even reading file footers for standard pruning is costly. Our inverted index maps values directly to file references, avoiding metadata scans entirely.
We often join a small predicate table with a massive base table. Previously, we used manual sideways information passing so our custom TableProvider could use the index during planning.
With DynamicFilters, this logic correctly moves to execution, but we lose the ability to trigger index-based pruning before files are opened.
The goal is to use runtime DynamicFilter values to query the inverted index and skip opening most files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org