You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "askalt (via GitHub)" <gi...@apache.org> on 2026/01/21 13:14:12 UTC
[I] Improve filter push-down [datafusion]
askalt opened a new issue, #19929:
URL: https://github.com/apache/datafusion/issues/19929
This issue covers two related filter push-down improvements.
## Pass previously pushed filters to supports_filters_pushdown
Currently, the optimization does not pass filters that were pushed in a previous run (`TableScan::filters`) to `TableProvider::supports_filters_pushdown(...)`.
If the optimizer runs multiple times, it may try to push filters into the table provider multiple times. In our DataFusion-based project, `supports_filters_pushdown(...)` has context-dependent behavior: the provider supports any single filter like `column = value`, but not multiple such filters at the same time.
Consider the following optimizer pipeline pattern:
1. Try to push `a = 1, b = 1`.
`supports_filters_pushdown` returns `[Exact, Inexact]`
OK: the optimizer records that `a = 1` is pushed and creates a filter node for `b = 1`.
...
Another optimization iteration.
2. Try to push b = 1.
`supports_filters_pushdown` returns `[Exact]`. Of course, the table provider can’t remember
all previously pushed filters, so it has no choice but to answer `Exact`.
Now, the optimizer thinks the conjunction `a = 1 AND b = 1` is supported exactly, but it is not.
To prevent this problem, I suggest passing filters that were already pushed into the scan earlier to `supports_filters_pushdown(...)`.
## Do not assume that filter support decision is stable
Consider the next scenario:
1. `supports_filters_pushdown` returns `Exact` on some filter, e.g. "a = 1", where column "a" is not
required by the query projection.
2. "a" is removed from the table provider projection by "optimize projection" rule.
3. `supports_filters_pushdown` changes a decision and returns `Inexact` on this filter the next time.
For example, input filters were changed and it prefers to use a new one.
4. "a" is not returned to the table provider projection which leads to filter that references a column which is
not a part of the schema.
Suggest to extend logic with the following actions:
1. Collect columns that are not used in the current table provider projection, but required for filter
expressions. Call it `additional_projection`.
2. If `additional_projection` is empty -- leave all as is.
3. Otherwise extend a table provider projection and wrap a plan with an additional projection node
to preserve schema used prior to this rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve filter push-down [datafusion]
Posted by "askalt (via GitHub)" <gi...@apache.org>.
askalt commented on issue #19929:
URL: https://github.com/apache/datafusion/issues/19929#issuecomment-3804046965
> > 2. Now, the optimizer thinks the conjunction `a = 1 AND b = 1` is supported exactly, but it is not.
>
> It sounds like the case will cause a wrong result, potentially, do you have a case for it?
In our DF based project we have a custom table provider that roughly can be described as a provider that supports only single filter exactly (reading from the tree-like index), but not a conjunction (internal engine can read from the single index at the moment), it is mocked here https://github.com/apache/datafusion/pull/20003/changes#diff-3fbd6073c97534f759a8c4cdab71bd506618972220eb58225c38cabf7061981bR4318
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve filter push-down [datafusion]
Posted by "askalt (via GitHub)" <gi...@apache.org>.
askalt commented on issue #19929:
URL: https://github.com/apache/datafusion/issues/19929#issuecomment-3780657126
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve filter push-down [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #19929:
URL: https://github.com/apache/datafusion/issues/19929#issuecomment-3807871343
> Another optimization iteration.
> Try to push b = 1.
> supports_filters_pushdown returns [Exact]. Of course, the table provider can’t remember
> all previously pushed filters, so it has no choice but to answer Exact.
> Now, the optimizer thinks the conjunction a = 1 AND b = 1 is supported exactly, but it is not.
What do you think about extending the "can pushdown filters" API somehow to include filters that are already planned to be pushed down?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve filter push-down [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #19929:
URL: https://github.com/apache/datafusion/issues/19929#issuecomment-3803967479
> 2\. Now, the optimizer thinks the conjunction `a = 1 AND b = 1` is supported exactly, but it is not.
It sounds like the case will cause a wrong result, potentially, do you have a case for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve filter push-down [datafusion]
Posted by "askalt (via GitHub)" <gi...@apache.org>.
askalt commented on issue #19929:
URL: https://github.com/apache/datafusion/issues/19929#issuecomment-3800948454
> I feel this is somewhat related to [#19894](https://github.com/apache/datafusion/issues/19894). I think we could avoid these sorts of issues if we accumulated filters on the `TableScan` node without modifying projections, etc. and only at the end of the optimizer run we do a single call into the `TableProvider` to sort out `Exact` filters and the final projection
Mentioned improvements could fix the artifacts, I pushed a small fix that just makes the rule itself idempotent regarding a table scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Improve filter push-down [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #19929:
URL: https://github.com/apache/datafusion/issues/19929#issuecomment-3780920113
I feel this is somewhat related to https://github.com/apache/datafusion/issues/19894. I think we could avoid these sorts of issues if we accumulated filters on the `TableScan` node without modifying projections, etc. and only at the end of the optimizer run we do a single call into the `TableProvider` to sort out `Exact` filters and the final projection
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org