You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/04/26 08:57:50 UTC
[GitHub] [hive] kasakrisz opened a new pull request, #3246: HIVE-26160: Materialized View rewrite does not check tables scanned in sub-query expressions
kasakrisz opened a new pull request, #3246:
URL: https://github.com/apache/hive/pull/3246
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Temporary apply the `HiveSubQueryRemoveRule` before collecting tables scanned by the initial CBO plan when performing sql text based MV rewrite.
### Why are the changes needed?
For Materialized View rewrite based on exact sql text match Hive uses the initial CBO plan which may contains sub-query expressions but the algorithm traverse the plan and collects the tables can not handle sub-query expressions.
### Does this PR introduce _any_ user-facing change?
Yes. If a query plan was rewritten to scan an outdated MV the results can be different.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -DskipSparkTests -Dtest=TestMiniLlapLocalCliDriver -Dqfile=materialized_view_rewrite_by_text_9.q -pl itests/qtest -Pitests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on a diff in pull request #3246: HIVE-26160: Materialized View rewrite does not check tables scanned in sub-query expressions
Posted by GitBox <gi...@apache.org>.
zabetak commented on code in PR #3246:
URL: https://github.com/apache/hive/pull/3246#discussion_r860040566
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java:
##########
@@ -2107,7 +2108,10 @@ private boolean isMaterializedViewRewritingByTextEnabled() {
}
private RelNode applyMaterializedViewRewritingByText(
- ASTNode queryToRewriteAST, RelNode originalPlan, RelOptCluster optCluster) {
+ ASTNode queryToRewriteAST,
+ RelNode originalPlan,
+ RelOptCluster optCluster,
+ RelMetadataProvider metadataProvider) {
Review Comment:
Thanks for looking more into this. There is certainly something weird around this but it is not a problem that we should investigate here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz merged pull request #3246: HIVE-26160: Materialized View rewrite does not check tables scanned in sub-query expressions
Posted by GitBox <gi...@apache.org>.
kasakrisz merged PR #3246:
URL: https://github.com/apache/hive/pull/3246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on a diff in pull request #3246: HIVE-26160: Materialized View rewrite does not check tables scanned in sub-query expressions
Posted by GitBox <gi...@apache.org>.
zabetak commented on code in PR #3246:
URL: https://github.com/apache/hive/pull/3246#discussion_r859794193
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java:
##########
@@ -2107,7 +2108,10 @@ private boolean isMaterializedViewRewritingByTextEnabled() {
}
private RelNode applyMaterializedViewRewritingByText(
- ASTNode queryToRewriteAST, RelNode originalPlan, RelOptCluster optCluster) {
+ ASTNode queryToRewriteAST,
+ RelNode originalPlan,
+ RelOptCluster optCluster,
+ RelMetadataProvider metadataProvider) {
Review Comment:
Is it necessary to add new `metadataProvider` parameter? Does `optCluster.getMetadataProvider()` return a different provider from the one passed as parameter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kasakrisz commented on a diff in pull request #3246: HIVE-26160: Materialized View rewrite does not check tables scanned in sub-query expressions
Posted by GitBox <gi...@apache.org>.
kasakrisz commented on code in PR #3246:
URL: https://github.com/apache/hive/pull/3246#discussion_r859840470
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java:
##########
@@ -2107,7 +2108,10 @@ private boolean isMaterializedViewRewritingByTextEnabled() {
}
private RelNode applyMaterializedViewRewritingByText(
- ASTNode queryToRewriteAST, RelNode originalPlan, RelOptCluster optCluster) {
+ ASTNode queryToRewriteAST,
+ RelNode originalPlan,
+ RelOptCluster optCluster,
+ RelMetadataProvider metadataProvider) {
Review Comment:
It returns a different provider.
[removeSubqueries](https://github.com/apache/hive/blob/7583142cbffcb3958a546a9aaa15700bbc243df9/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java#L2395) could also get a provider from the plan in it's 1st param like
```
basePlan.getCluster().getMetadataProvider()
```
However this does not returns the same instance passed in the 2nd parameter.
One provider instance has 14 entries the other one has 22.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org