You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Steve Carlin (Code Review)" <ge...@cloudera.org> on 2026/01/24 01:13:15 UTC
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Steve Carlin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/23900
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query option RUNTIME_PLANNERS. The four choices for
runtime planners are:
ORIGINAL (default): Use the Original planner for queries.
CALCITE: Use the Calcite planner for queries.
CALCITE,ORIGINAL: Use the Calcite planner. If the Calcite planner fails
during compilation, use the Original planner.
ORIGINAL,CALCITE : Use the Original planner. If the Original planner fails
during compilation, use the Calcite planner.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of whether CALCITE or
CALCITE,ORIGINAL is set.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The CALCITE,ORIGINAL
option is set if the --use_calcite_planner=true server option is
specified. This will allow easier transitioning to the Calcite planner
because the user will assured that even if there is a compilation failure
in Calcite, their exisitng query will still run with the original planner.
In the query options module, the code to handle multiple TPlannerTypes is
similar to the ENABLED_RUNTIME_FILTER_TYPES option. However, the
RUNTIME_PLANNERS option had to be in a list, not a set, so some additional
code was needed to support that.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/planner/CalcitePlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/runtime_planners.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
13 files changed, 255 insertions(+), 75 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/4
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 7:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23900/7/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/23900/7/fe/src/main/java/org/apache/impala/service/Frontend.java@2437
PS7, Line 2437: // None of the planners worked. Exceptional case: If only Calcite, fallback for unsupported SQLs
line too long (100 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jan 2026 17:58:27 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 10:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/21624/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Feb 2026 23:09:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 13: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 13
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:35:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Fang-Yu Rao, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23900
to look at the new patch set (#12).
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query options PLANNER and FALLBACK_PLANNER which are
of type TPlannerType
The 3 options for TPlannerType are:
ORIGINAL (default) : Use the Original planner for queries.
CALCITE : Use the Calcite planner for queries.
NONE : Convenient option for setting the FALLBACK_PLANNER.
This should never be used for the PLANNER option.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of what is chosen for
FALLBACK_PLANNER.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The PLANNER=CALCITE
and FALLBACK_PLANNER=ORIGINAL options are set if the
--use_calcite_planner=true server option is specified. This will allow
easier transitioning to the Calcite planner because the user will be
assured that even if there is a compilation failure in Calcite,
their existing query will still run with the original planner.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/fallback_planner.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
A tests/query_test/test_calcite_planner.py
13 files changed, 355 insertions(+), 71 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/12
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 12
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Fang-Yu Rao, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23900
to look at the new patch set (#13).
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query options PLANNER and FALLBACK_PLANNER which are
of type TPlannerType
The 3 options for TPlannerType are:
ORIGINAL (default) : Use the Original planner for queries.
CALCITE : Use the Calcite planner for queries.
NONE : Convenient option for setting the FALLBACK_PLANNER.
This should never be used for the PLANNER option.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of what is chosen for
FALLBACK_PLANNER.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The PLANNER=CALCITE
and FALLBACK_PLANNER=ORIGINAL options are set if the
--use_calcite_planner=true server option is specified. This will allow
easier transitioning to the Calcite planner because the user will be
assured that even if there is a compilation failure in Calcite,
their existing query will still run with the original planner.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/fallback_planner.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
A tests/query_test/test_calcite_planner.py
13 files changed, 354 insertions(+), 72 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/13
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 13
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 7:
A little bit of an overhaul based on the feedback of the comments. The query options are changed to two separate planner options (planner, fallback) whereas previously it was one query option in a comma separated list
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jan 2026 17:58:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Fang-Yu Rao, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23900
to look at the new patch set (#15).
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query options PLANNER and FALLBACK_PLANNER which are
of type TPlannerType
The 3 options for TPlannerType are:
ORIGINAL (default) : Use the Original planner for queries.
CALCITE : Use the Calcite planner for queries.
NONE : Convenient option for setting the FALLBACK_PLANNER.
This should never be used for the PLANNER option.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of what is chosen for
FALLBACK_PLANNER.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The PLANNER=CALCITE
and FALLBACK_PLANNER=ORIGINAL options are set if the
--use_calcite_planner=true server option is specified. This will allow
easier transitioning to the Calcite planner because the user will be
assured that even if there is a compilation failure in Calcite,
their existing query will still run with the original planner.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/fallback_planner.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
A tests/query_test/test_calcite_planner.py
13 files changed, 353 insertions(+), 72 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/15
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 15
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23900/9/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/9/common/thrift/ImpalaService.thrift@1028
PS9, Line 1028: USE_CALCITE_PLANNER = 191 // Removed
We have REMOVED_QUERY_OPT_FN, can we use that on this option in query-options.h? That's what seems to have been done to others marked with "// Removed".
http://gerrit.cloudera.org:8080/#/c/23900/9/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/9/common/thrift/Query.thrift@a1164
PS9, Line 1164:
nit: unnecessary whitespace change
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py
File tests/custom_cluster/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py@41
PS9, Line 41: @CustomClusterTestSuite.with_args(start_args="--env_vars=USE_CALCITE_PLANNER=true")
Does this env_var still do something? It shows up a few places, can we clean it up?
If we no longer need to specify additional options to enable USE_CALCITE_PLANNER, these could also be moved out of custom cluster tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Feb 2026 21:46:59 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 11:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21644/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 11
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:37:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/impala-server.cc@2030
PS5, Line 2030: if (FLAGS_use_calcite_planner) {
> Maybe use_calcite_as_default_planner?
I was thinking more like having the startup param specify a string value equivalent to the query option rather than a boolean. (It would need to be subject to the same validation.)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc@1396
PS5, Line 1396: case TImpalaQueryOptions::RUNTIME_PLANNERS: {
: std::vector<TPlannerType::type> runtime_planners;
: const string planners = std::regex_replace(value, std::regex("^\"|\"$"), "");
: vector<string> str_types;
: split(str_types, planners, is_any_of(","), token_compress_on);
: TrimAndRemoveEmptyString(str_types);
: for (const auto& t : str_types) {
: TPlannerType::type planner_type;
: RETURN_IF_ERROR(GetThriftEnum(t, "planner type",
: _TPlannerType_VALUES_TO_NAMES, &planner_type));
: runtime_planners.push_back(planner_type);
: }
: query_options->__set_runtime_planners(runtime_planners);
> It doesn't, but this might be tricky?
If "runtime_planners" is empty when you are about to set it on query_options, return a not-OK status. I'm not sure if that is actually possible, but I think we have a precondition in the frontend requiring the list to be non-empty.
There may be a distinction between empty string (which has length 0) and empty quotes (length 2 with two quote characters).
We can do a std::find(runtime_planners.begin(), runtime_planners.end(), planner_type) != runtime_planners.end() for checking that we are not adding a duplicate.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Jan 2026 01:09:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc@1396
PS5, Line 1396: case TImpalaQueryOptions::RUNTIME_PLANNERS: {
: std::vector<TPlannerType::type> runtime_planners;
: const string planners = std::regex_replace(value, std::regex("^\"|\"$"), "");
: vector<string> str_types;
: split(str_types, planners, is_any_of(","), token_compress_on);
: TrimAndRemoveEmptyString(str_types);
: for (const auto& t : str_types) {
: TPlannerType::type planner_type;
: RETURN_IF_ERROR(GetThriftEnum(t, "planner type",
: _TPlannerType_VALUES_TO_NAMES, &planner_type));
: runtime_planners.push_back(planner_type);
: }
: query_options->__set_runtime_planners(runtime_planners);
> If "runtime_planners" is empty when you are about to set it on query_option
Don't mind doing this, but it does break precedence. I think this is the code that handles it right now:
https://github.com/apache/impala/blob/master/be/src/service/query-options.cc#L266-L271
We would have to special case it (or maybe change it to "if (value.empty()) and optionHandlesEmpty())
Lemme know what you think.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Jan 2026 15:33:06 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/impala-server.cc@2030
PS5, Line 2030: if (FLAGS_use_calcite_planner) {
> Should we change the use_calcite_planner flag to something else?
Maybe use_calcite_as_default_planner?
Or is that too wordy?
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jan 2026 23:53:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23900
to look at the new patch set (#5).
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query option RUNTIME_PLANNERS. The four choices for
runtime planners are:
ORIGINAL (default): Use the Original planner for queries.
CALCITE: Use the Calcite planner for queries.
CALCITE,ORIGINAL: Use the Calcite planner. If the Calcite planner fails
during compilation, use the Original planner.
ORIGINAL,CALCITE : Use the Original planner. If the Original planner fails
during compilation, use the Calcite planner.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of whether CALCITE or
CALCITE,ORIGINAL is set.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The CALCITE,ORIGINAL
option is set if the --use_calcite_planner=true server option is
specified. This will allow easier transitioning to the Calcite planner
because the user will assured that even if there is a compilation failure
in Calcite, their exisitng query will still run with the original planner.
In the query options module, the code to handle multiple TPlannerTypes is
similar to the ENABLED_RUNTIME_FILTER_TYPES option. However, the
RUNTIME_PLANNERS option had to be in a list, not a set, so some additional
code was needed to support that.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/planner/CalcitePlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/runtime_planners.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
13 files changed, 257 insertions(+), 75 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/5
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23900/5/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/5/common/thrift/ImpalaService.thrift@1033
PS5, Line 1033: RUNTIME_PLANNERS = 191
Enum item USE_CALCITE_PLANNER=191 changed to RUNTIME_PLANNERS=191 in TImpalaQueryOptions. This might break the compatibility between impalad and catalogd/statestore during upgrade
http://gerrit.cloudera.org:8080/#/c/23900/5/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/5/common/thrift/Query.thrift@790
PS5, Line 790: 192: optional list<PlanNodes.TPlannerType> runtime_planners = [TPlannerType.ORIGINAL]
Renaming field 'use_calcite_planner' to 'runtime_planners' in TQueryOptions might break the compatibility between impalad and catalogd/statestore during upgrade
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Jan 2026 15:43:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 15:
Ok, sorry about that...I had a slight worry after I submitted the last fix and I wanted to make sure it was ok.
So first: Apologies: I tested all the other tests, but I forgot about the ranger one and sigh, that was the one test that failed.
The reason it failed: The failed version had 2 lines to set configuration options for the test:
non_owner_client.set_configuration({"planner": "CALCITE"})
non_owner_client.set_configuration({"fallback_planner": "CALCITE"})
I had thought that each of these lines would be setting the configuration parameters. But the second "set_configuration" overwrote the first one. after the second one, the "planner" went back to the default "ORIGINAL" planner (the fallback_planner was CALCITE) and that was why the test failed.
I could also have fixed this by combining the 2 query options into the set_configuration call. But this fix is fine too. Even though the fallback_planner is ORIGINAL, this isn't a test about whether fallback works properly, it's a test that ensures that fallback is happening.
In other words: with or without the fallback planner option, this will fallback to the original planner due to the UnsupportedFeatureException being thrown, so the test is working as designed.
If you can give this a +1 (sorry, again!), I'd appreciate it!
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 15
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Feb 2026 01:47:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query options PLANNER and FALLBACK_PLANNER which are
of type TPlannerType
The 3 options for TPlannerType are:
ORIGINAL (default) : Use the Original planner for queries.
CALCITE : Use the Calcite planner for queries.
NONE : Convenient option for setting the FALLBACK_PLANNER.
This should never be used for the PLANNER option.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of what is chosen for
FALLBACK_PLANNER.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The PLANNER=CALCITE
and FALLBACK_PLANNER=ORIGINAL options are set if the
--use_calcite_planner=true server option is specified. This will allow
easier transitioning to the Calcite planner because the user will be
assured that even if there is a compilation failure in Calcite,
their existing query will still run with the original planner.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/fallback_planner.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
12 files changed, 304 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/9
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23900
to look at the new patch set (#8).
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query options PLANNER and FALLBACK_PLANNER which are
of type TPlannerType
The 3 options for TPlannerType are:
ORIGINAL (default) : Use the Original planner for queries.
CALCITE : Use the Calcite planner for queries.
NONE : Convenient option for setting the FALLBACK_PLANNER.
This should never be used for the PLANNER option.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of what is chosen for
FALLBACK_PLANNER.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The PLANNER=CALCITE
and FALLBACK_PLANNER=ORIGINAL options are set if the
--use_calcite_planner=true server option is specified. This will allow
easier transitioning to the Calcite planner because the user will be
assured that even if there is a compilation failure in Calcite,
their existing query will still run with the original planner.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/planner/CalcitePlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/fallback_planner.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
13 files changed, 306 insertions(+), 74 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/8
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 8
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/23900/5//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/23900/5//COMMIT_MSG@9
PS5, Line 9: RUNTIME_PLANNERS
I'd rather not include "runtime" in the name. Maybe "planner_precedence" or "planner_preference"? I don't know a good name for this.
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/impala-server.cc@2030
PS5, Line 2030: if (FLAGS_use_calcite_planner) {
Should we change the use_calcite_planner flag to something else?
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc@1396
PS5, Line 1396: case TImpalaQueryOptions::RUNTIME_PLANNERS: {
: std::vector<TPlannerType::type> runtime_planners;
: const string planners = std::regex_replace(value, std::regex("^\"|\"$"), "");
: vector<string> str_types;
: split(str_types, planners, is_any_of(","), token_compress_on);
: TrimAndRemoveEmptyString(str_types);
: for (const auto& t : str_types) {
: TPlannerType::type planner_type;
: RETURN_IF_ERROR(GetThriftEnum(t, "planner type",
: _TPlannerType_VALUES_TO_NAMES, &planner_type));
: runtime_planners.push_back(planner_type);
: }
: query_options->__set_runtime_planners(runtime_planners);
Does this produce an error if runtime_planners is empty? It's possible to add a test in query-options-test.cc (e.g. there is a test there for enabled_runtime_filter_types).
Also, we can verify that there are no duplicates (e.g. CALCITE,CALCITE,CALCITE).
http://gerrit.cloudera.org:8080/#/c/23900/5/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/5/common/thrift/ImpalaService.thrift@1033
PS5, Line 1033: RUNTIME_PLANNERS = 191
> Enum item USE_CALCITE_PLANNER=191 changed to RUNTIME_PLANNERS=191 in TImpal
I think you can leave USE_CALCITE_PLANNER, but remove the comment and add a "// Removed" comment next to it.
Then add a new number for this one. That should fix this complaint.
http://gerrit.cloudera.org:8080/#/c/23900/5/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/5/common/thrift/Query.thrift@790
PS5, Line 790: 192: optional list<PlanNodes.TPlannerType> runtime_planners = [TPlannerType.ORIGINAL]
> Renaming field 'use_calcite_planner' to 'runtime_planners' in TQueryOptions
I think it is ok to remove use_calcite_planner and leave this particular numeric id unused. So, this new option can be a new numeric id, and the check shouldn't complain.
http://gerrit.cloudera.org:8080/#/c/23900/5/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/23900/5/fe/src/main/java/org/apache/impala/service/Frontend.java@2406
PS5, Line 2406: private TExecRequest getTExecRequestWithFallback(
: PlanCtx planCtx, EventSequence timeline) throws ImpalaException {
: TExecRequest request = null;
: LOG.info("Searching for planner to use...");
: TPlannerType plannerType = getPlannerType(planCtx, false);
: CompilerFactory compilerFactory = getCompilerFactory(plannerType);
: LOG.info("Attempting with " + compilerFactory.getPlannerString() + " Planner.");
: Throwable firstAttemptPlannerException = null;
: TPlannerType fallbackPlanner = null;
I guess the question is whether this code should be explicit about having at most two options or whether it should try to be handling the list as a list. This is a gist of what the list one would look like:
bool onlyCalcite = plannerList.size() == 1 && plannerList.get(0) == CALCITE;
Throwable error = null;
TQueryOptions queryOptions = ctx.getQueryContext().client_request.getQuery_options();
List<TPlannerType> plannerTypes = queryOptions.getRuntime_planners();
for (TPlannerType plannerType : plannerTypes) {
error = null;
CompilerFactory compilerFactory = getCompilerFactory(plannerType);
try {
TExecRequest request = getTExecRequest(compilerFactory, planCtx, timeline);
addPlannerToProfile(compiler.getPlannerString());
return request;
} catch (Throwable e) {
error = e;
// Add entry for compilerFactory.getPlannerString() + "FailureReason" with the exception info
}
}
// None of the planners worked. Exceptional case: If only Calcite, fallback for unsupported SQLs
if (error != null && onlyCalcite && isUnsupportedCalciteSQL(planCtx, error)) {
// run original planner
}
throw error;
http://gerrit.cloudera.org:8080/#/c/23900/5/fe/src/main/java/org/apache/impala/service/Frontend.java@2423
PS5, Line 2423: // For Calcite planner: Certain queries aren't supported by the planner, e.g. DDL.
: // If the query isn't supported, we always fallback to the original planner,
: // regardless of whether the RUNTIME_PLANNERS query option specifies a fallback
: // planner.
: fallbackPlanner = isUnsupportedCalciteSQL(plannerType, planCtx, e)
: ? TPlannerType.ORIGINAL
: : getPlannerType(planCtx, true);
We only need to run this if this is Calcite and there isn't a fallback planner, so we can get the fallback planner and only do this isUnsupportedCalciteSQL check if fallbackPlanner is null. That would avoid an extra parse for CALCITE,ORIGINAL.
http://gerrit.cloudera.org:8080/#/c/23900/5/fe/src/main/java/org/apache/impala/service/Frontend.java@3822
PS5, Line 3822: try {
: CompilerFactory calciteFactory = (CompilerFactory) Class.forName(
: "org.apache.impala.calcite.service." +
: "CalciteCompilerFactory").newInstance();
: if (calciteFactory != null) {
: LOG.info("Found Calcite Planner, using it.");
: return calciteFactory;
: } else {
: LOG.info("Could not find Calcite planner.");
: //XXX: what kind of exception should be thrown here? Or should we fallback
: // even though explicit planner is mentioned?
: throw new RuntimeException("Could not find Calcite Planner");
: }
: } catch (Exception e) {
: LOG.info("Could not find Calcite planner: " + e);
: //XXX: what kind of exception should be thrown here? Or should we fallback even
: // though explicit planner is mentioned?
: throw new RuntimeException(e);
: }
This basically shouldn't happen and is a configuration issue. Users always have a way out: Don't use Calcite. So, I don't feel like this really needs to fall back or be handled.
I also think we could fail startup if someone has Calcite enabled by default and we can't find Calcite.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jan 2026 23:21:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/12/tests/query_test/test_calcite_planner.py
File tests/query_test/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/12/tests/query_test/test_calcite_planner.py@43
PS12, Line 43: """Calcite planner does not work in local catalog mode yet."""
How is this comment relevant?
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 12
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:24:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 14:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13079/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 14
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:41:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 11:
(2 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23900/11/tests/query_test/test_calcite_planner.py
File tests/query_test/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/11/tests/query_test/test_calcite_planner.py@20
PS11, Line 20: import pytest
flake8: F401 'pytest' imported but unused
http://gerrit.cloudera.org:8080/#/c/23900/11/tests/query_test/test_calcite_planner.py@50
PS11, Line 50: class TestFallbackPlanner(ImpalaTestSuite):
flake8: E302 expected 2 blank lines, found 1
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 11
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:09:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 15:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/13080/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 15
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Feb 2026 01:00:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 13:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21646/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 13
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:59:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc@1396
PS5, Line 1396: case TImpalaQueryOptions::RUNTIME_PLANNERS: {
: std::vector<TPlannerType::type> runtime_planners;
: const string planners = std::regex_replace(value, std::regex("^\"|\"$"), "");
: vector<string> str_types;
: split(str_types, planners, is_any_of(","), token_compress_on);
: TrimAndRemoveEmptyString(str_types);
: for (const auto& t : str_types) {
: TPlannerType::type planner_type;
: RETURN_IF_ERROR(GetThriftEnum(t, "planner type",
: _TPlannerType_VALUES_TO_NAMES, &planner_type));
: runtime_planners.push_back(planner_type);
: }
: query_options->__set_runtime_planners(runtime_planners);
> How about a "planner" query option and a "fallback_planner" query option? I
Oh, I like that! Ok, I'll code it that way.
Default will be original for both.
Will upload soon.
Thanks!
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jan 2026 05:03:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Fang-Yu Rao, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23900
to look at the new patch set (#10).
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query options PLANNER and FALLBACK_PLANNER which are
of type TPlannerType
The 3 options for TPlannerType are:
ORIGINAL (default) : Use the Original planner for queries.
CALCITE : Use the Calcite planner for queries.
NONE : Convenient option for setting the FALLBACK_PLANNER.
This should never be used for the PLANNER option.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of what is chosen for
FALLBACK_PLANNER.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The PLANNER=CALCITE
and FALLBACK_PLANNER=ORIGINAL options are set if the
--use_calcite_planner=true server option is specified. This will allow
easier transitioning to the Calcite planner because the user will be
assured that even if there is a compilation failure in Calcite,
their existing query will still run with the original planner.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/fallback_planner.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
12 files changed, 305 insertions(+), 73 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/10
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/12/tests/custom_cluster/test_calcite_planner.py
File tests/custom_cluster/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/12/tests/custom_cluster/test_calcite_planner.py@43
PS12, Line 43: """Calcite planner does not work in local catalog mode yet."""
We use local catalog by default now. I think this comment can be removed.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 12
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:25:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23900/9/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/9/common/thrift/ImpalaService.thrift@1028
PS9, Line 1028: USE_CALCITE_PLANNER = 191 // Removed
> We have REMOVED_QUERY_OPT_FN, can we use that on this option in query-optio
Done
http://gerrit.cloudera.org:8080/#/c/23900/9/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/9/common/thrift/Query.thrift@a1164
PS9, Line 1164:
> nit: unnecessary whitespace change
Done
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py
File tests/custom_cluster/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py@41
PS9, Line 41: @CustomClusterTestSuite.with_args(start_args="--env_vars=USE_CALCITE_PLANNER=true")
> Does this env_var still do something? It shows up a few places, can we clea
I suppose this is up for debate...
I purposefully left this here because I think it's kinda nice to be able to explicitly use the calcite planner via a direct server option. I know that default query options can also be overridden as an option, so perhaps this is redundantly redundant.
So I'm up in the air about this one. If it were my preference, I'd leave it, but if consensus is that it should be removed, I wouldn't put up much of an argument about it.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Feb 2026 22:53:49 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has restored this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Restored
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: restore
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has abandoned this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 14: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/13079/
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 14
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 23:28:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 12:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py
File tests/custom_cluster/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py@41
PS9, Line 41: @CustomClusterTestSuite.with_args(start_args="--env_vars=USE_CALCITE_PLANNER=true")
> Got it, that makes sense.
That makes sense.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 12
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:12:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 12:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21645/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 12
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:37:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21490/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Jan 2026 16:55:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23900
to look at the new patch set (#7).
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query options PLANNER and FALLBACK_PLANNER which are
of type TPlannerType
The 3 options for TPlannerType are:
ORIGINAL (default) : Use the Original planner for queries.
CALCITE : Use the Calcite planner for queries.
NONE : Convenient option for setting the FALLBACK_PLANNER.
This should never be used for the PLANNER option.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of what is chosen for
FALLBACK_PLANNER.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The PLANNER=CALCITE
and FALLBACK_PLANNER=ORIGINAL options are set if the
--use_calcite_planner=true server option is specified. This will allow
easier transitioning to the Calcite planner because the user will be
assured that even if there is a compilation failure in Calcite,
their existing query will still run with the original planner.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/ExecRequestCreator.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/planner/CalcitePlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/fallback_planner.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
13 files changed, 305 insertions(+), 74 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/7
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21619/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Feb 2026 20:29:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 10:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23900/10/be/src/service/query-options.h
File be/src/service/query-options.h:
http://gerrit.cloudera.org:8080/#/c/23900/10/be/src/service/query-options.h@377
PS10, Line 377: REMOVED_QUERY_OPT_FN(use_calcite_planner, USE_CALCITE_PLANNER, \
: TQueryOptionLevel::ADVANCED) \
REMOVED_QUERY_OPT_FN doesn't take a third arg, so we'll need to remove the TQueryOptionLevel::ADVANCED.
http://gerrit.cloudera.org:8080/#/c/23900/10/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/23900/10/fe/src/main/java/org/apache/impala/service/Frontend.java@3800
PS10, Line 3800: @Nullable
: private CompilerFactory getCompilerFactory(TPlannerType plannerType) {
: switch (plannerType) {
: case ORIGINAL:
: return new CompilerFactoryImpl();
: case CALCITE: {
: try {
: CompilerFactory calciteFactory = (CompilerFactory) Class.forName(
: "org.apache.impala.calcite.service." +
: "CalciteCompilerFactory").newInstance();
: if (calciteFactory != null) {
: LOG.info("Found Calcite Planner, using it.");
: return calciteFactory;
: } else {
: LOG.info("Could not find Calcite planner.");
: throw new RuntimeException("Could not find Calcite Planner");
: }
: } catch (Exception e) {
: LOG.info("Could not find Calcite planner: " + e);
: throw new RuntimeException(e);
: }
: }
: default:
: throw new RuntimeException("Unknown planner type: [" + plannerType + "]");
: }
: }
When https://gerrit.cloudera.org/#/c/23935/ goes in, this code will change.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Wed, 11 Feb 2026 17:25:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 13: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 13
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:37:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 15:
I just kicked off a rerun, but hold off on a re-review until I check one more thing and then I'll explain the change.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 15
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Feb 2026 01:03:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 4:
(6 comments)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/23900/4/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/4/common/thrift/ImpalaService.thrift@1033
PS4, Line 1033: RUNTIME_PLANNERS = 191
Enum item USE_CALCITE_PLANNER=191 changed to RUNTIME_PLANNERS=191 in TImpalaQueryOptions. This might break the compatibility between impalad and catalogd/statestore during upgrade
http://gerrit.cloudera.org:8080/#/c/23900/4/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/4/common/thrift/Query.thrift@790
PS4, Line 790: 192: optional list<PlanNodes.TPlannerType> runtime_planners = [TPlannerType.ORIGINAL]
Renaming field 'use_calcite_planner' to 'runtime_planners' in TQueryOptions might break the compatibility between impalad and catalogd/statestore during upgrade
http://gerrit.cloudera.org:8080/#/c/23900/4/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/23900/4/fe/src/main/java/org/apache/impala/service/Frontend.java@2423
PS4, Line 2423: // For Calcite planner: Certain queries aren't supported by the planner, e.g. DDL. If
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/23900/4/fe/src/main/java/org/apache/impala/service/Frontend.java@2462
PS4, Line 2462: // If an exception was thrown, it failed to parse in the original planner, so there is
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/23900/4/fe/src/main/java/org/apache/impala/service/Frontend.java@3830
PS4, Line 3830: //XXX: what kind of exception should be thrown here? Or should we fallback even
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/23900/4/tests/custom_cluster/test_calcite_planner.py
File tests/custom_cluster/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/4/tests/custom_cluster/test_calcite_planner.py@50
PS4, Line 50: class TestRuntimePlannersPlanner(CustomClusterTestSuite):
flake8: E302 expected 2 blank lines, found 1
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 24 Jan 2026 01:15:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23900/4/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/4/common/thrift/ImpalaService.thrift@1033
PS4, Line 1033: RUNTIME_PLANNERS = 191
> Enum item USE_CALCITE_PLANNER=191 changed to RUNTIME_PLANNERS=191 in TImpal
This is ok. CALCITE_PLANNER is still pre-beta mode and nobody should be using it.
http://gerrit.cloudera.org:8080/#/c/23900/4/common/thrift/Query.thrift
File common/thrift/Query.thrift:
http://gerrit.cloudera.org:8080/#/c/23900/4/common/thrift/Query.thrift@790
PS4, Line 790: 192: optional list<PlanNodes.TPlannerType> runtime_planners = [TPlannerType.ORIGINAL]
> Renaming field 'use_calcite_planner' to 'runtime_planners' in TQueryOptions
This is ok. CALCITE_PLANNER is still pre-beta mode and nobody should be using it.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Mon, 26 Jan 2026 15:42:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21540/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 7
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jan 2026 19:01:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query options PLANNER and FALLBACK_PLANNER which are
of type TPlannerType
The 3 options for TPlannerType are:
ORIGINAL (default) : Use the Original planner for queries.
CALCITE : Use the Calcite planner for queries.
NONE : Convenient option for setting the FALLBACK_PLANNER.
This should never be used for the PLANNER option.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of what is chosen for
FALLBACK_PLANNER.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The PLANNER=CALCITE
and FALLBACK_PLANNER=ORIGINAL options are set if the
--use_calcite_planner=true server option is specified. This will allow
easier transitioning to the Calcite planner because the user will be
assured that even if there is a compilation failure in Calcite,
their existing query will still run with the original planner.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Reviewed-on: http://gerrit.cloudera.org:8080/23900
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/fallback_planner.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
A tests/query_test/test_calcite_planner.py
13 files changed, 353 insertions(+), 72 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Michael Smith: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 16
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 15: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 15
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Feb 2026 18:56:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc@1396
PS5, Line 1396: case TImpalaQueryOptions::RUNTIME_PLANNERS: {
: std::vector<TPlannerType::type> runtime_planners;
: const string planners = std::regex_replace(value, std::regex("^\"|\"$"), "");
: vector<string> str_types;
: split(str_types, planners, is_any_of(","), token_compress_on);
: TrimAndRemoveEmptyString(str_types);
: for (const auto& t : str_types) {
: TPlannerType::type planner_type;
: RETURN_IF_ERROR(GetThriftEnum(t, "planner type",
: _TPlannerType_VALUES_TO_NAMES, &planner_type));
: runtime_planners.push_back(planner_type);
: }
: query_options->__set_runtime_planners(runtime_planners);
> Don't mind doing this, but it does break precedence. I think this is the c
Here is what I mean: If the local variable "runtime_planners" is empty when you are about to set it in query_options on line 1408, return an error.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Jan 2026 21:24:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 11:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/23900/10/be/src/service/query-options.h
File be/src/service/query-options.h:
http://gerrit.cloudera.org:8080/#/c/23900/10/be/src/service/query-options.h@377
PS10, Line 377: REMOVED_QUERY_OPT_FN(use_calcite_planner, USE_CALCITE_PLANNER) \
: QUERY_OPT_FN(json_binary_format, JSON_BINARY_FORMAT, TQueryOptionLevel::REGULAR) \
> REMOVED_QUERY_OPT_FN doesn't take a third arg, so we'll need to remove the
Ugh...as you can prolly tell, I didn't compile before submitting because I thought this was such a simple change.
My bad :(
Fixed now (and compiled :) )
http://gerrit.cloudera.org:8080/#/c/23900/10/fe/src/main/java/org/apache/impala/service/Frontend.java
File fe/src/main/java/org/apache/impala/service/Frontend.java:
http://gerrit.cloudera.org:8080/#/c/23900/10/fe/src/main/java/org/apache/impala/service/Frontend.java@3800
PS10, Line 3800: throw new TransactionException(e.getMessage());
: } finally {
: // Call KuduTransaction.close() explicitly to stop sending automatic
: // keepalive messages by the 'txn' handle.
: txn.close();
: }
: }
: }
:
: private CompilerFactory getCompilerFactory(TPlannerType plannerType) {
: switch (plannerType) {
: case ORIGINAL:
: LOG.info("Using Original Planner.");
: return ORIGINAL_COMPILER_FACTORY;
: case CALCITE:
: if (CALCITE_COMPILER_FACTORY == null) {
: LOG.info("Could not find Calcite planner.");
: throw new RuntimeException("Could not find Calcite Planner");
: }
: LOG.info("Using Calcite Planner.");
: return CALCITE_COMPILER_FACTORY;
: default:
: throw new RuntimeException("Unknown planner type: [" + plannerType + "]");
: }
: }
:
> When https://gerrit.cloudera.org/#/c/23935/ goes in, this code will change.
Noted and merged (and changed slightly)
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py
File tests/custom_cluster/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py@41
PS9, Line 41: @CustomClusterTestSuite.with_args(start_args="--env_vars=USE_CALCITE_PLANNER=true")
> We can deal with it separately. But the other cases don't need to be custom
Got it, that makes sense.
I created a new test under query_tests. In that test, I have all the same tests and they just use the query options.
I did keep this file to test the server flag and only kept the one test.
Does it make sense to do it this way? Or should I minimize the custom cluster test even more?
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 11
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:11:55 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 9:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/23900/12/tests/custom_cluster/test_calcite_planner.py
File tests/custom_cluster/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/12/tests/custom_cluster/test_calcite_planner.py@43
PS12, Line 43: """Calcite planner does not work in local catalog mode yet."""
> We use local catalog by default now. I think this comment can be removed.
Done
http://gerrit.cloudera.org:8080/#/c/23900/12/tests/query_test/test_calcite_planner.py
File tests/query_test/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/12/tests/query_test/test_calcite_planner.py@43
PS12, Line 43:
> How is this comment relevant?
Done
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 9
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:31:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 15:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21651/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 15
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Feb 2026 01:24:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Fang-Yu Rao, Joe McDonnell, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/23900
to look at the new patch set (#11).
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
IMPALA-14488: Calcite planner: Support fallback to Original planner
Introduced the query options PLANNER and FALLBACK_PLANNER which are
of type TPlannerType
The 3 options for TPlannerType are:
ORIGINAL (default) : Use the Original planner for queries.
CALCITE : Use the Calcite planner for queries.
NONE : Convenient option for setting the FALLBACK_PLANNER.
This should never be used for the PLANNER option.
The Calcite planner currently does not support DDL and other various
queries (e.g. complex types). If a known unsupported query is detected,
the original planner will be used regardless of what is chosen for
FALLBACK_PLANNER.
The fallback planner will allow for easier transition for users who
want to use the new Calcite planner without having to worry about
backward compatibility issues that the compiler doesn't support.
One example that is currently not supported (in runtime_planners.test)
is the "ignore nulls" phrase which needs to be inside the parens for
Impala, but outside the parens for the Calcite parser.
At the time of this commit, there are also various compiler exceptions
thrown for Calcite which are not yet supported. The PLANNER=CALCITE
and FALLBACK_PLANNER=ORIGINAL options are set if the
--use_calcite_planner=true server option is specified. This will allow
easier transitioning to the Calcite planner because the user will be
assured that even if there is a compilation failure in Calcite,
their existing query will still run with the original planner.
Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
---
M be/src/service/impala-server.cc
M be/src/service/query-options.cc
M be/src/service/query-options.h
M common/thrift/ImpalaService.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Query.thrift
M fe/src/main/java/org/apache/impala/service/Frontend.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/CalcitePlannerTest.java
M java/calcite-planner/src/test/java/org/apache/impala/calcite/planner/TpcdsCpuCostPlannerTest.java
A testdata/workloads/functional-query/queries/QueryTest/fallback_planner.test
M tests/authorization/test_ranger.py
M tests/custom_cluster/test_calcite_planner.py
A tests/query_test/test_calcite_planner.py
13 files changed, 355 insertions(+), 71 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/23900/11
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 11
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21482/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Sat, 24 Jan 2026 02:26:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc@1396
PS5, Line 1396: case TImpalaQueryOptions::RUNTIME_PLANNERS: {
: std::vector<TPlannerType::type> runtime_planners;
: const string planners = std::regex_replace(value, std::regex("^\"|\"$"), "");
: vector<string> str_types;
: split(str_types, planners, is_any_of(","), token_compress_on);
: TrimAndRemoveEmptyString(str_types);
: for (const auto& t : str_types) {
: TPlannerType::type planner_type;
: RETURN_IF_ERROR(GetThriftEnum(t, "planner type",
: _TPlannerType_VALUES_TO_NAMES, &planner_type));
: runtime_planners.push_back(planner_type);
: }
: query_options->__set_runtime_planners(runtime_planners);
> Does this produce an error if runtime_planners is empty? It's possible to a
It doesn't, but this might be tricky?
I actually tried this while testing. If I set it to empty, it goes back to the default ORIGINAL. I suppose that didn't apply to enabled_runtime_filter_types which is empty by default (prolly?)
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Jan 2026 23:55:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py
File tests/custom_cluster/test_calcite_planner.py:
http://gerrit.cloudera.org:8080/#/c/23900/9/tests/custom_cluster/test_calcite_planner.py@41
PS9, Line 41: @CustomClusterTestSuite.with_args(start_args="--env_vars=USE_CALCITE_PLANNER=true")
> I suppose this is up for debate...
We can deal with it separately. But the other cases don't need to be custom cluster tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 10
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Feb 2026 23:02:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 14: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 14
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 12 Feb 2026 18:41:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 15: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 15
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Feb 2026 05:43:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc@1396
PS5, Line 1396: case TImpalaQueryOptions::RUNTIME_PLANNERS: {
: std::vector<TPlannerType::type> runtime_planners;
: const string planners = std::regex_replace(value, std::regex("^\"|\"$"), "");
: vector<string> str_types;
: split(str_types, planners, is_any_of(","), token_compress_on);
: TrimAndRemoveEmptyString(str_types);
: for (const auto& t : str_types) {
: TPlannerType::type planner_type;
: RETURN_IF_ERROR(GetThriftEnum(t, "planner type",
: _TPlannerType_VALUES_TO_NAMES, &planner_type));
: runtime_planners.push_back(planner_type);
: }
: query_options->__set_runtime_planners(runtime_planners);
> Ok, just tested and there is indeed a difference between empty and ""
How about a "planner" query option and a "fallback_planner" query option? If they different and the first one fails, it uses the fallback_planner.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jan 2026 02:00:21 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc@1396
PS5, Line 1396: case TImpalaQueryOptions::RUNTIME_PLANNERS: {
: std::vector<TPlannerType::type> runtime_planners;
: const string planners = std::regex_replace(value, std::regex("^\"|\"$"), "");
: vector<string> str_types;
: split(str_types, planners, is_any_of(","), token_compress_on);
: TrimAndRemoveEmptyString(str_types);
: for (const auto& t : str_types) {
: TPlannerType::type planner_type;
: RETURN_IF_ERROR(GetThriftEnum(t, "planner type",
: _TPlannerType_VALUES_TO_NAMES, &planner_type));
: runtime_planners.push_back(planner_type);
: }
: query_options->__set_runtime_planners(runtime_planners);
> Here is what I mean: If the local variable "runtime_planners" is empty when
Ok, just tested and there is indeed a difference between empty and ""
I can do what you are saying and catch "" and reject it. It would be different behavior from empty string though which is caught at line 289 in this method (266 before changes). Wouldn't we want the same behavior? If so, I can change line 289 to do what you are saying, but that's what I meant above about "breaking precedence" because I'd have to put special case functionality for that.
This whole thing can also be avoided if I leave use_calcite_planner alone and add a boolean for fallback as well. This logic wouldn't be scalable for a third planner, but do we need to be scalable for this? I'm starting to lean towards doing it that way since it won't have this quote problem. What do you think?
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Thu, 29 Jan 2026 23:49:30 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/21541/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 8
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jan 2026 18:26:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14488: Calcite planner: Support fallback to Original planner
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/23900 )
Change subject: IMPALA-14488: Calcite planner: Support fallback to Original planner
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc
File be/src/service/query-options.cc:
http://gerrit.cloudera.org:8080/#/c/23900/5/be/src/service/query-options.cc@1396
PS5, Line 1396: case TImpalaQueryOptions::RUNTIME_PLANNERS: {
: std::vector<TPlannerType::type> runtime_planners;
: const string planners = std::regex_replace(value, std::regex("^\"|\"$"), "");
: vector<string> str_types;
: split(str_types, planners, is_any_of(","), token_compress_on);
: TrimAndRemoveEmptyString(str_types);
: for (const auto& t : str_types) {
: TPlannerType::type planner_type;
: RETURN_IF_ERROR(GetThriftEnum(t, "planner type",
: _TPlannerType_VALUES_TO_NAMES, &planner_type));
: runtime_planners.push_back(planner_type);
: }
: query_options->__set_runtime_planners(runtime_planners);
> Oh, I like that! Ok, I'll code it that way.
Yes, original for both makes sense. If someone switches to calcite, they get fallback unless they specifically turn it off. The fallback planner could have a NONE option as a clearer way to express turning it off (but original/original has no fallback and calcite/calcite has no fallback).
--
To view, visit http://gerrit.cloudera.org:8080/23900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id1fdc5ef92fff84e89af0e19c4246cc15e2ea823
Gerrit-Change-Number: 23900
Gerrit-PatchSet: 5
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <sc...@cloudera.com>
Gerrit-Comment-Date: Fri, 30 Jan 2026 16:42:12 +0000
Gerrit-HasComments: Yes