You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Stamatis Zampetakis (Code Review)" <ge...@cloudera.org> on 2026/09/03 09:15:04 UTC
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Stamatis Zampetakis has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24787
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
The rewrite algorithm relies on the MaterializedViewRule hierarchy. In order to perform a rewrite the MV rules use the following RelMetadataQuery APIs:
* RelMetadataQuery#getNodeTypes
* RelMetadataQuery#getAllPredicates
* RelMetadataQuery#getExpressionLineage
* RelMetadataQuery#getTableReferences
However, the standard BuiltInMetadata handlers provided by Calcite don't recognize the ImpalaCTEConsumer operator so once it is introduced in the plan by one rewrite all the rest that fall in the same RelSubset are nuked.
The commit adds customised handlers for returing the appropriate metadata when encountering ImpalaCTEConsumer allowing further rewrites to trigger properly and explore the entire search space.
Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
---
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdExpressionLineage.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdNodeTypes.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdTableReferences.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMetadataProvider.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test
7 files changed, 2,855 insertions(+), 2,315 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/24787/1
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 1
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23923/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 3
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Sep 2026 13:01:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Stamatis Zampetakis (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24787
to look at the new patch set (#3).
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
The rewrite algorithm relies on the MaterializedViewRule hierarchy. In
order to perform a rewrite the MV rules use the following
RelMetadataQuery APIs:
* RelMetadataQuery#getNodeTypes
* RelMetadataQuery#getAllPredicates
* RelMetadataQuery#getExpressionLineage
* RelMetadataQuery#getTableReferences
However, the standard BuiltInMetadata handlers provided by Calcite don't
recognize the ImpalaCTEConsumer operator so once it is introduced in the
plan by one rewrite all the rest that fall in the same RelSubset are
nuked.
The commit adds customised handlers for returning the appropriate
metadata when encountering ImpalaCTEConsumer allowing further rewrites
to trigger properly and explore the entire search space.
Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
---
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdExpressionLineage.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdNodeTypes.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdTableReferences.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMetadataProvider.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test
7 files changed, 2,856 insertions(+), 2,315 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/24787/3
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 3
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Stamatis Zampetakis (Code Review)" <ge...@cloudera.org>.
Stamatis Zampetakis has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 3:
(3 comments)
At the moment the problem is visible only on Q75 cause many other larger CTEs in TPC-DS queries are not rewritable (IMPALA-15334) so they are pruned for other reasons.
http://gerrit.cloudera.org:8080/#/c/24787/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24787/2//COMMIT_MSG@9
PS2, Line 9: The rewrite algorithm relies on the MaterializedViewRule hierarchy. In
> I updated some commit formatting and fixed a typo.
Great, thanks!
http://gerrit.cloudera.org:8080/#/c/24787/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java:
http://gerrit.cloudera.org:8080/#/c/24787/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java@36
PS2, Line 36: return mq.getAllPredicates(r.getCTE());
> Can you add a comment somewhere why returning null is the appropriate actio
I pushed a new patch where the new metadata handlers use another Calcite interface and thus all these generic catch-all methods returning null were removed.
FYI: Returning null from a metadata handler means I don't know how to handle this expression (no metadata available). These generic/catch all methods trigger if we don't have a registered handler for an expression (e.g., ImpalaCteProducer). Now that the catch all methods were removed from the ImpalaRelMd* handlers if we ever encounter an unsupported/unregistered RelNode subclass we will use Calcite's built-in catch all methods.
http://gerrit.cloudera.org:8080/#/c/24787/2/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test:
http://gerrit.cloudera.org:8080/#/c/24787/2/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test@80
PS2, Line 80: Per-Host Resource Estimates: Memory=199.45GB
> Unrelated, I've seen the CTE plans seem to increase the per-host resource e
Ack
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 3
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Comment-Date: Fri, 04 Sep 2026 12:51:06 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 2:
(3 comments)
This really only updates the plan for 1 query?
http://gerrit.cloudera.org:8080/#/c/24787/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/24787/2//COMMIT_MSG@9
PS2, Line 9: The rewrite algorithm relies on the MaterializedViewRule hierarchy. In
I updated some commit formatting and fixed a typo.
http://gerrit.cloudera.org:8080/#/c/24787/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java:
http://gerrit.cloudera.org:8080/#/c/24787/2/java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java@36
PS2, Line 36: return null;
Can you add a comment somewhere why returning null is the appropriate action for this and the other handlers?
http://gerrit.cloudera.org:8080/#/c/24787/2/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test:
http://gerrit.cloudera.org:8080/#/c/24787/2/testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test@80
PS2, Line 80: Per-Host Resource Estimates: Memory=199.45GB
Unrelated, I've seen the CTE plans seem to increase the per-host resource estimates. We should look into that at some point.
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 2
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Sep 2026 16:25:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has uploaded a new patch set (#2) to the change originally created by Stamatis Zampetakis. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
The rewrite algorithm relies on the MaterializedViewRule hierarchy. In
order to perform a rewrite the MV rules use the following
RelMetadataQuery APIs:
* RelMetadataQuery#getNodeTypes
* RelMetadataQuery#getAllPredicates
* RelMetadataQuery#getExpressionLineage
* RelMetadataQuery#getTableReferences
However, the standard BuiltInMetadata handlers provided by Calcite don't
recognize the ImpalaCTEConsumer operator so once it is introduced in the
plan by one rewrite all the rest that fall in the same RelSubset are
nuked.
The commit adds customised handlers for returning the appropriate
metadata when encountering ImpalaCTEConsumer allowing further rewrites
to trigger properly and explore the entire search space.
Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
---
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdExpressionLineage.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdNodeTypes.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdTableReferences.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMetadataProvider.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test
7 files changed, 2,855 insertions(+), 2,315 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/24787/2
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 2
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 4
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Sep 2026 21:51:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 3:
Ran all 103 TpcdsCtePlannerTest cases on PS3, along with CalciteOptimizerTest and TestImpalaRelMdNonCumulativeCost: all passed.
Both CTE query tests (test_cte_plans and test_cte_distributed) also passed on three impalads with planner and fallback_planner set to CALCITE.
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 3
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Comment-Date: Sun, 06 Sep 2026 21:55:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 4
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Sep 2026 17:22:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Aleksandr Efimov (Code Review)" <ge...@cloudera.org>.
Aleksandr Efimov has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 3: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 3
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Comment-Date: Sun, 06 Sep 2026 10:04:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/23909/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 1
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Sep 2026 09:40:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13993/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 4
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Sep 2026 17:22:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 3
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Sep 2026 17:21:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/24787 )
Change subject: IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
......................................................................
IMPALA-15327: CTE rewrite algorithm cannot explore all suggestions
The rewrite algorithm relies on the MaterializedViewRule hierarchy. In
order to perform a rewrite the MV rules use the following
RelMetadataQuery APIs:
* RelMetadataQuery#getNodeTypes
* RelMetadataQuery#getAllPredicates
* RelMetadataQuery#getExpressionLineage
* RelMetadataQuery#getTableReferences
However, the standard BuiltInMetadata handlers provided by Calcite don't
recognize the ImpalaCTEConsumer operator so once it is introduced in the
plan by one rewrite all the rest that fall in the same RelSubset are
nuked.
The commit adds customised handlers for returning the appropriate
metadata when encountering ImpalaCTEConsumer allowing further rewrites
to trigger properly and explore the entire search space.
Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Reviewed-on: http://gerrit.cloudera.org:8080/24787
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdAllPredicates.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdExpressionLineage.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdNodeTypes.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMdTableReferences.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaRelMetadataProvider.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds_cte/tpcds-q75.test
7 files changed, 2,856 insertions(+), 2,315 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24787
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic32d11cd82e2484944b6d03d5fb56900e8e1b7a4
Gerrit-Change-Number: 24787
Gerrit-PatchSet: 5
Gerrit-Owner: Stamatis Zampetakis <za...@gmail.com>
Gerrit-Reviewer: Aleksandr Efimov <dz...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Stamatis Zampetakis <za...@gmail.com>