You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2024/07/04 08:45:22 UTC
[I] Reorder filter predicates according to evaluation cost [datafusion]
Dandandan opened a new issue, #11262:
URL: https://github.com/apache/datafusion/issues/11262
### Is your feature request related to a problem or challenge?
After https://github.com/apache/datafusion/pull/11247 is merged we can look at ordering the boolean expressions according to a measure of evaluation cost.
### Describe the solution you'd like
We can reorder expressions:
E.g. a expression like the following:
`URL LIKE '%google%' AND code = 404`.
Likely would be better reordered to `code = 404 AND URL LIKE '%google%' ` in order to benefit most from short circuiting.
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity [datafusion]
Posted by "neilconway (via GitHub)" <gi...@apache.org>.
neilconway commented on issue #11262:
URL: https://github.com/apache/datafusion/issues/11262#issuecomment-4477086594
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #11262: Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity
URL: https://github.com/apache/datafusion/issues/11262
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11262:
URL: https://github.com/apache/datafusion/issues/11262#issuecomment-4481598087
@neilconway I wonder what you were thinking of tackling?
IMO the version of this that #22144 is trying to tackle necessarily needs to live external to the expression itself. It's much more tightly wound with the parquet scan itself because we aren't just deciding the order in which filters are evaluated but also where the IO happens (eager vs. late materialization).
It could still make sense to have `BinaryExpr` do reordering itself. For example we are not going to split / track `id = 1 OR message ilike '%foo%'` as separate filters during scans. And there are still `FilterExec`s, join conditions, other places where binary expressions are used.
I was also wondering if a "flattened" binary expression that can do a better job of re-using buffers, etc. would make sense. It seems necessary to do some sort of re-ordering sanely (otherwise you have to re-build the expression tree which would be hard once the query is executing).
Another issue that appears is the "what is an expression anyway" issue. Various places remap children / rewrite the expression in ways. It's not always clear when the expression is the same (should share selectivity tracking) or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11262:
URL: https://github.com/apache/datafusion/issues/11262#issuecomment-2210727263
We could potentially do some simple heuristics that would catch the common case -- like "treat regexp as very slow and do them after other predicates"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11262:
URL: https://github.com/apache/datafusion/issues/11262#issuecomment-4481377491
Related to
- https://github.com/apache/datafusion/pull/22144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #11262:
URL: https://github.com/apache/datafusion/issues/11262#issuecomment-2208623885
> I've seen discussions about predicate reordering in the calcite community before, and one of the big problems is that the engine doing reordering of predicates invalidates the user-designed order of predicates, if the user understands that our short circuit optimisation writes the sql as a better order, but the engine reordering invalidates his efforts.
Good call, if we do it, it needs to be configurable so users/engines can disable the optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity [datafusion]
Posted by "suibianwanwank (via GitHub)" <gi...@apache.org>.
suibianwanwank commented on issue #11262:
URL: https://github.com/apache/datafusion/issues/11262#issuecomment-2208607983
I've seen discussions about predicate reordering in the calcite community before, and one of the big problems is that the engine doing reordering of predicates invalidates the user-designed order of predicates, if the user understands that our short circuit optimisation writes the sql as a better order, but the engine reordering invalidates his efforts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Reorder boolean expressions (including filter predicates) according to evaluation cost / selectivity [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #11262:
URL: https://github.com/apache/datafusion/issues/11262#issuecomment-4481375678
I think @adriangb has had some more recent ideas on this (to more dynamically re-order expressions based on actual observed selectivity rather than trying to heuristically update them):
- https://github.com/apache/datafusion/pull/22236
ALso possibly related
- https://github.com/apache/datafusion/pull/22237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org