You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kosiew (via GitHub)" <gi...@apache.org> on 2026/05/07 04:59:01 UTC
[I] [Optimizer] Consolidate repeated filter-rebuild patterns in PushDownFilter [datafusion]
kosiew opened a new issue, #22056:
URL: https://github.com/apache/datafusion/issues/22056
## Summary
`PushDownFilter` rebuilds filter nodes in many branch-specific paths. This duplication increases maintenance cost and makes rule invariants harder to keep consistent.
## Background and Motivation
#21667 introduced `make_filter` to centralize filter construction, but push/keep/reinsert logic remains duplicated across plan variants.
Today, each branch tends to repeat the same shape:
1. split predicates into pushable/keep sets
2. rebuild one or more child filters
3. reinsert kept predicates above
4. return transformed plan
This pattern appears in `Sort`, `Distinct`, `Repartition`, `Projection`, `Union`, `Extension`, `Aggregate`, `Window`, `Unnest`, and join-related paths.
## Problem Statement
Filter reconstruction mechanics are duplicated across many branches in `datafusion/optimizer/src/push_down_filter.rs`.
Concrete symptoms:
- repeated `make_filter(...)` and `Arc::new(...)` call patterns
- repeated split/push/keep control flow
- repeated branch-local child replacement logic
## Why This Matters
- Correctness risk: duplicated rewrite logic makes subtle behavior divergence more likely across plan-node branches.
- Invariant drift: filter reconstruction/reinsertion conventions are harder to enforce when spread across many sites.
- Review burden: future changes require auditing many branches for equivalent behavior.
- Evolvability: adding new single-input node rewrites repeats boilerplate and invites copy-paste defects.
## Proposed Direction
Introduce a small, internal unary-node helper for filter reconstruction within `PushDownFilter`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Optimizer] Consolidate repeated filter-rebuild patterns in PushDownFilter [datafusion]
Posted by "SubhamSinghal (via GitHub)" <gi...@apache.org>.
SubhamSinghal commented on issue #22056:
URL: https://github.com/apache/datafusion/issues/22056#issuecomment-4412742025
@kosiew can I help here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Optimizer] Consolidate repeated filter-rebuild patterns in PushDownFilter [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on issue #22056:
URL: https://github.com/apache/datafusion/issues/22056#issuecomment-4667787861
@SubhamSinghal
It's yours 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org