You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/04/26 13:19:11 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue #98: [Rust] Add constant folding to expressions during logically planning
alamb opened a new issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98
*Note*: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-9770
The high level idea is that if an expression can be partially evaluated during planning time then
# The execution time will be increased
# There may be additional optimizations possible (like removing entire LogicalPlan nodes, for example)
I recently saw the following selection expression created (by the [predicate push down|https://github.com/apache/arrow/pull/7880])
{code}
Selection: #a Eq Int64(1) And #b GtEq Int64(1) And #a LtEq Int64(1) And #a Eq Int64(1) And #b GtEq Int64(1) And #a LtEq Int64(1)
TableScan: test projection=None
{code}
This could be simplified significantly:
1. Duplicate clauses could be removed (e.g. `#a Eq Int64(1) And #a Eq Int64(1)` --> `#a Eq Int64(1)`)
2. Algebraic simplification (e.g. if `A<=B and A=5`, is the same as `A=5`)
Inspiration can be taken from the postgres code that evaluates constant expressions https://doxygen.postgresql.org/clauses_8c.html#ac91c4055a7eb3aa6f1bc104479464b28
(in this case, for example if you have a predicate A=5 then you can basically substitute in A=5 for any expression higher up in the the plan).
Other classic optimizations include things such as `A OR TRUE` --> `A`, `A AND TRUE` --> TRUE, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98#issuecomment-826828967
Comment from Jorge Leitão(jorgecarleitao) @ 2020-08-23T21:51:08.856+0000:
<pre>That is a cool idea. It seems to be a case of algebraic simplification. Maybe there is already a rust library that can do that work for us?</pre>
Comment from Remi Dettai(rdettai) @ 2020-12-08T14:11:11.412+0000:
<pre>This could also be used to apply filter pushdown into the catalog, if the combination (and) of:
- the file/partition statistics ({{col( x ) > min and col( x ) < max}})
- the filter expression
can be simplified to {{false}}, their is no need to read that file/partition. Isn't that wonderful ? :)
Note: there is an implementation of this folding for the C++ dataset API: [https://github.com/apache/arrow/blob/master/cpp/src/arrow/dataset/filter.h]</pre>
Comment from Andrew Lamb(alamb) @ 2021-04-26T12:33:27.229+0000:
<pre>Migrated to github: https://github.com/apache/arrow-rs/issues/96</pre>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] rdettai commented on issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
rdettai commented on issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98#issuecomment-909256340
What is missing from https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/optimizer/constant_folding.rs to close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98#issuecomment-909284494
I don't think anything else is needed to close this issue. We can always open a new issue to improve part of the folding. Thanks for the spot @rdettai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] rdettai commented on issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
rdettai commented on issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98#issuecomment-909256340
What is missing from https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/optimizer/constant_folding.rs to close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] rdettai commented on issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
rdettai commented on issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98#issuecomment-909256340
What is missing from https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/optimizer/constant_folding.rs to close this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
alamb closed issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
alamb closed issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98#issuecomment-909284494
I don't think anything else is needed to close this issue. We can always open a new issue to improve part of the folding. Thanks for the spot @rdettai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98#issuecomment-909284494
I don't think anything else is needed to close this issue. We can always open a new issue to improve part of the folding. Thanks for the spot @rdettai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #98: [Rust] Add constant folding to expressions during logically planning
Posted by GitBox <gi...@apache.org>.
alamb closed issue #98:
URL: https://github.com/apache/arrow-datafusion/issues/98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org