You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/06/10 16:24:04 UTC
[I] Adaptive predicate evaluation [datafusion]
alamb opened a new issue, #22883:
URL: https://github.com/apache/datafusion/issues/22883
## Summary
Decide *where* each filter conjunct runs (pushed into the Parquet decoder as a row filter, applied post-scan in memory, or dropped) by measuring its selectivity and throughput at runtime, instead of fixing the placement statically at plan time.
## Context
Today, with `pushdown_filters = true`, DataFusion pushes all predicates into the Parquet reader as row filters and orders them with the static `reorder_filters` heuristic, which can't react to data skew and pays an extra IO round-trip per filter column even when a filter prunes almost nothing. Because row filters change the IO pattern (read filter columns → mask → read projection), the right placement depends on each filter's *measured* selectivity, which is only known at runtime. This issue tracks an adaptive cost model that tracks per-filter effectiveness across files/row-groups and continuously promotes, demotes, or drops conjuncts.
## Related PRs
- #22144 — [Experiment] Adaptive filter pushdown (umbrella experiment)
- #22234 — `OptionalFilterPhysicalExpr` wrapper + proto (mark a filter as droppable)
- #22235 — Per-conjunct pruning statistics for `PruningPredicate`
- #22236 — `SelectivityTracker` adaptive filter cost model
- #22237 — Adaptive filter pushdown for the parquet scan (integration)
- #21752 — Adaptive filter scheduling for Parquet scans (prior full PR)
- #20363 — (Test) Advanced adaptive filter selectivity evaluation
- #19639 — feat: adaptive filter selectivity tracking for Parquet row filters (closed prototype)
- apache/arrow-rs#9659 — complementary compute-only filter evaluation optimizations
## Related issues
- #15512 — [Epic] Dynamic filtering related items
- #21207 — [DISCUSSION] Future of Dynamic Filters Sync
- #3463 — Enable parquet `filter_pushdown` by default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5812471634
@adriangb that looks very nice, I'll catch up on it after holidays
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-4761349372
I think the next step here is https://github.com/apache/datafusion/pull/22698. It introduces the substrate for stats tracking and adaptivity. I plan to first use this with `FilterExec` then re-use it for scans.
@neilconway or @xudong963 would one/both of you like to help with this work (review or pushing he features)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5868874755
> Its strongest idea is separating two questions: whether a filter is required for correctness, and whether evaluating it is worthwhile on the current data.
Yes, I agree formalizing this notion would be a very nice improvement
> My preference is to use the wrapper for this experimental stack. For a longer-lived public mechanism, I would favor FilterConjunct once the expression and its metadata can travel through rewrites as one unit, without relying solely on positional reassociation.
@xudong963 I don't quite follow what you are saying here. Are you saying you prefer https://github.com/apache/datafusion/pull/25760 ?
I also reviewed (and really like) the `FilterConjunct` / `PhysicalFilter` API in
- https://github.com/apache/datafusion/pull/25760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5869036689
I found https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt has a lot of information and I didn't fully digest it all, however I think pretty much the whole thing relies on being able to tell required from optional filters, so I agree that is a good first area to focus on
Once we sort that out, then we can work on dynamically deciding to enable/disable each filter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-4771155000
I will try and review
- https://github.com/apache/datafusion/pull/22698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5869003098
I think https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt has another good observation that is not directly part of optional filters, specifically the DynamicFilter representation of a hash table in a Join.
> Other engines use a cheaper, weaker filter when the build side is large. DataFusion does the opposite.
That might be worth pulling out into its own discussion issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5808424593
I've been experimenting with a way to handle "optional" filters, meaning filters that are not needed for correctness, such as hash join / TopK dynamic filters and prune-only copies. Design notes, including how Trino, DuckDB, ClickHouse, Velox and Spark handle this: https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt
In short: an `Optional(...)` wrapper marks the filter. A consumer may skip it only when it finds it on the root AND chain. Per-stream gates pause optional filters that remove too few rows. `PushedDown` keeps its two states.
This is an experiment. The PRs are drafts and stacked:
- #25673 `OptionalFilterPhysicalExpr` + `split_optional` + proto
- #25681 hash join / TopK / aggregate mark their dynamic filters optional (only EXPLAIN changes)
- #25674 `OptionalFilterGate` + `datafusion.execution.optional_filter_mode` (`always` default | `adaptive` | `pruning_only`)
- #25682 the Parquet scan uses the mode for optional row filters
- #25683 `FilterExec`: skip optional conjuncts, plus a smaller take on adaptive reordering (#22698 stays open for its history)
Independent of the stack but related:
- #25677 collapse partitioned InList join filters into one `IN` list (from #24235)
- #25670 docs: what `PushedDown::No` means (why no `Inexact` state)
- #22384 rebased
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5868936738
Another thing I was thinking about was "could this mechanism work for dynamically choosing parquet pushdown filters".
Today we have to choose between
```text
(filter)
(repartition)
(scan)
```
and full pushdown
```
(scan, filter)
```
However, with optional filters we could potentially model it like two filters that switched at runtime between optional and required. Something like:
Plan starts like
```
(filter = OPTIONAL)
(repartition)
(scan, filter = REQUIRED)
```
And if we need to stop evaluating during the scan we could switch the pattern to
```
(filter = REQUIRED)
(repartition)
(scan, filter = OPTIONAL)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5902432849
> [@xudong963](https://github.com/xudong963) I looked into positional re-association. The conclusion I came to is that it is a pre-existing footgun in `main` (e.g. you reply `Yes` for the wrong filter and it's dropped). There is no good way to solve it, ultimately consumers need to manipulate these filters. No API is both restrictive enough to prevent errors but also flexible enough to accommodate all consumers. Using an `id` of sorts to track filters doesn't really solve it either. So I'd say let's not worry about it for now, wdyt?
Thanks for looking into this. That makes sense: the positional contract already exists for PushedDown, so I agree that solving it generally should not block #25760.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-4774544245
Nice, I'll help review the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5861859580
Thanks for working on this. I read the full [Optional Filters design document](https://claude.ai/artifact/SSz7t6hPyhFWp1MDPecVqt). **Overall, the direction is worth pursuing and the document is very clear, love it**
Its strongest idea is separating two questions: whether a filter is required for correctness, and whether evaluating it is worthwhile on the current data.
Having the producer mark a filter as `Optional` and the consumer make the runtime decision gives each side a clear responsibility. -- very clear design and abstract;
Allowing a filter to be skipped only when it appears on the root `AND` chain also addresses the correctness risks posed by `NOT`, `OR`, and other expression contexts.
I would raise some points in review:
1. **Make plan properties an explicit correctness prerequisite.** A scan that uses a predicate only for statistics pruning must not claim that every output row satisfies it. The recently reported [#25779](https://github.com/apache/datafusion/issues/25779) demonstrates that an invalid equivalence claim can produce wrong results for `ORDER BY … LIMIT`;
2. The latest design has moved beyond the simulator’s pass-ratio threshold to comparing evaluation cost with downstream savings. However, downstream savings for TopK and aggregate filters still rely on an estimate;
3. **Resolve the API choice.** The `Optional(...)` expression wrapper limits the initial changes, but it also enters expression rewriting, serialization, and plan-property paths. [#25760](https://github.com/apache/datafusion/pull/25760) proposes storing optionality in a `FilterConjunct` instead. My preference is to use the wrapper for this experimental stack. For a longer-lived public mechanism, I would favor FilterConjunct once the expression and its metadata can travel through rewrites as one unit, without relying solely on positional reassociation.
I see there are many existing PRs related to the topic; could you list them in order? That way, it'll be easier for everyone else to track and review them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5881609874
> [@xudong963](https://github.com/xudong963) I don't quite follow what you are saying here. Are you saying you prefer [#25760](https://github.com/apache/datafusion/pull/25760) ?
For the API we ultimately merge, I lean toward the FilterConjunct / PhysicalFilter approach in #25760. Optionality seems to be a property of a conjunct at a particular plan site, rather than a property of PhysicalExpr.
I meant that the wrapper is a low-churn way to prototype the adaptive behavior on a draft branch, not that we should merge it as the public API and migrate later. My remaining concern with #25760 is the positional reassociation of optional flags through expression-only pushdown interfaces: I would like to make sure a rewrite or reorder cannot attach a flag to the wrong expression. Overall, I like the direction of #25760.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Adaptive predicate evaluation [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #22883:
URL: https://github.com/apache/datafusion/issues/22883#issuecomment-5893406097
@xudong963 I looked into positional re-association. The conclusion I came to is that it is a pre-existing footgun in `main` (e.g. you reply `Yes` for the wrong filter and it's dropped). There is no good way to solve it, ultimately consumers need to manipulate these filters. No API is both restrictive enough to prevent errors but also flexible enough to accommodate all consumers. Using an `id` of sorts to track filters doesn't really solve it either. So I'd say let's not worry about it for now, wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org