You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2026/01/22 23:00:56 UTC
[I] [EPIC] Siumplify datetime predicates using "preimage" [datafusion]
alamb opened a new issue, #19946:
URL: https://github.com/apache/datafusion/issues/19946
### Is your feature request related to a problem or challenge?
It is much easier for optimizers to reason about predicates of the form `<col> op <constant> `expressions. They often can't optimize anywhere near as well if they have a scalar function wrapping them
This includes DataFusion's [PruningPredicate](https://docs.rs/datafusion/latest/datafusion/physical_optimizer/pruning/struct.PruningPredicate.html#contains-analysis-and-minmax-rewrite)
For example the predicate looking for a particular year
```sql
WHERE EXTRACT (YEAR FROM k) = 2024
```
Can be rewritten as
```sql
k >= 2024-01-01 AND k < 2025-01-01.
```
And then k is easier to pushdown and subject to range analysis, etc.
The ClickHouse paper : https://www.vldb.org/pvldb/vol17/p3731-schulze.pdf calles these "preimage" (from the [mathematical term](https://en.wikipedia.org/wiki/Image_(mathematics))) for this rewrite (I think toYear(k) is the equivalent of EXTRACT(YEAR from k))
> Second, some functions can compute the preimage of a given function result. This is used to replace comparisons of constants with function calls on the key columns by comparing the key column value with the preimage. For example, toYear(k) = 2024 can be replaced by k >= 2024-01-01 && k < 2025-01-01.
This ticket tracks adding such optimziations to DataFusion
### Describe the solution you'd like
- [ ] https://github.com/apache/datafusion/issues/18320
- [ ] https://github.com/apache/datafusion/issues/18319
- [ ] https://github.com/apache/datafusion/issues/19889
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Simplify datetime predicates using "preimages" [datafusion]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #19946:
URL: https://github.com/apache/datafusion/issues/19946#issuecomment-3825773959
I have a follow-up PR to #18320
- #20051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org