You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/18 03:44:04 UTC
[GitHub] [arrow-datafusion] jackwener opened a new issue, #4267: [EPIC]: reimplement all rule which contains global-state
jackwener opened a new issue, #4267:
URL: https://github.com/apache/arrow-datafusion/issues/4267
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Current some rule contains `global-state`
Such as `limit_push_down` use `ancestor: &Ancestor` collect the limit-state of whole tree.
- #4266
- #4265
- #4264
- #4263
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jackwener commented on issue #4267: [EPIC]: reimplement all rules which contains global-state
Posted by GitBox <gi...@apache.org>.
jackwener commented on issue #4267:
URL: https://github.com/apache/arrow-datafusion/issues/4267#issuecomment-1319546486
After finish this job,
I will need to finish #4208 .
After finish them, All preparations will be done.
I will do #3972 ! It will make our optimizer simple and easy-use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #4267: [EPIC]: reimplement all rules which contains global-state
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #4267:
URL: https://github.com/apache/arrow-datafusion/issues/4267#issuecomment-1320222716
👍 the idea of pattern matching on the way back up during rewrite rules makes lots of sense to me 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] mingmwang commented on issue #4267: [EPIC]: reimplement all rules which contains global-state
Posted by GitBox <gi...@apache.org>.
mingmwang commented on issue #4267:
URL: https://github.com/apache/arrow-datafusion/issues/4267#issuecomment-1328884927
I think it depends on what the global-state is. If the global states are predicates or something related to Column/Exprs, then we should remove such kind of global states.. But for some optimization rules, it might be better to keep a global state so that it is easily to figure out a best plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb closed issue #4267: [EPIC]: reimplement all rules which contains global-state
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #4267: [EPIC]: reimplement all rules which contains global-state
URL: https://github.com/apache/arrow-datafusion/issues/4267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jackwener commented on issue #4267: [EPIC]: reimplement all rules which contains global-state
Posted by GitBox <gi...@apache.org>.
jackwener commented on issue #4267:
URL: https://github.com/apache/arrow-datafusion/issues/4267#issuecomment-1319545030
let me explain why.
Due to global-state exists, we can't just optimize a subtree, we must traverse the whole tree.
In fact, I prefer to `Pattern-match`, we match a subtree `pattern`, and just optimize the pattren/subtree.
A example: Merge Limit
```rust
match plan {
Limit => {
match childPlan => {
Limit => { merge(limit, childlimit) }
_ => { don't match pattern, return None or recurse....}
}
}
_ => { don't match pattern, return None or recurse....}
}
```
we just match `limit-limit`, and handle it.
BUT, global-state force we must handle various situations， and we must traverse the tree all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jackwener commented on issue #4267: [EPIC]: reimplement all rules which contains global-state
Posted by GitBox <gi...@apache.org>.
jackwener commented on issue #4267:
URL: https://github.com/apache/arrow-datafusion/issues/4267#issuecomment-1329338179
> I think it depends on what the global-state is. If the global states are predicates or something related to Column/Exprs, then we should remove such kind of global states.. But for some optimization rules, it might be better to keep a global state so that it is easily to figure out a best plan.
Agree it, `global-state` of all rules that I change is related `Column/Exprs` or `limit number`.
> I think today in DataFusion's logical optimizer rules, most of the complexity comes from dealing with predicates.
Agree, especially for rule about projection/filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] mingmwang commented on issue #4267: [EPIC]: reimplement all rules which contains global-state
Posted by GitBox <gi...@apache.org>.
mingmwang commented on issue #4267:
URL: https://github.com/apache/arrow-datafusion/issues/4267#issuecomment-1328893809
I think today in DataFusion's logical optimizer rules, most of the complexity come from dealing with predicates. Let's why I want to introduce the QueryConstraints into the logical plan structs.
https://github.com/apache/arrow-datafusion/issues/4222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jackwener commented on issue #4267: [EPIC]: reimplement all rules which contains global-state
Posted by GitBox <gi...@apache.org>.
jackwener commented on issue #4267:
URL: https://github.com/apache/arrow-datafusion/issues/4267#issuecomment-1319764479
cc @andygrove @alamb @mingmwang @liukun4515
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org