You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Steve Carlin (Code Review)" <ge...@cloudera.org> on 2025/01/24 16:35:22 UTC
[Impala-ASF-CR] IMPALA-13520: Support in clause coercing
Steve Carlin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22103
Change subject: IMPALA-13520: Support in clause coercing
......................................................................
IMPALA-13520: Support in clause coercing
Calcite has special processing for any in clause. It has a callback
function that allows all the parameters to be coerced into its proper
type. While we do have a mechanism to do coercion, it only handles
functions, and the in clause is handled specially within Calcite.
So we use the Calcite mechanism to derive a common Impala type and
coerce all the parameters.
The CombineValueNodesRule is also needed for this change. There is a
test case in test_exprs.test where an in clause contains 10,000
params in side the IN clause (e.g. int_col IN (1, 2, 3, ..., 10000).
In this case, Calcite creates 10,000 Values RelNodes which takes way
too long to process on the execution side. The rule combines all
the Values RelNodes into one Values RelNode with many tuples, which
Impala handles quickly when converted into the physical Impala
PlanNode.
Change-Id: I310d3d8deb489e097dd58ef8f8bc63aefe118051
---
A java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/CombineValueNodesRule.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeCoercionFactory.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeCoercionImpl.java
5 files changed, 259 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/22103/3
--
To view, visit http://gerrit.cloudera.org:8080/22103
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I310d3d8deb489e097dd58ef8f8bc63aefe118051
Gerrit-Change-Number: 22103
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
[Impala-ASF-CR] IMPALA-13520: Support in clause coercing
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22103 )
Change subject: IMPALA-13520: Support in clause coercing
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18102/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22103
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I310d3d8deb489e097dd58ef8f8bc63aefe118051
Gerrit-Change-Number: 22103
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 27 Jan 2025 21:54:56 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-13520: Support in clause coercing
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Steve Carlin has abandoned this change. ( http://gerrit.cloudera.org:8080/22103 )
Change subject: IMPALA-13520: Support in clause coercing
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/22103
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I310d3d8deb489e097dd58ef8f8bc63aefe118051
Gerrit-Change-Number: 22103
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-13520: Support in clause coercing
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22103 )
Change subject: IMPALA-13520: Support in clause coercing
......................................................................
Patch Set 3:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22103/3/java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/CombineValueNodesRule.java
File java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/CombineValueNodesRule.java:
http://gerrit.cloudera.org:8080/#/c/22103/3/java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/CombineValueNodesRule.java@60
PS3, Line 60: ImmutableList.Builder<ImmutableList<RexLiteral>> rowBuilder = new ImmutableList.Builder();
line too long (94 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/22103
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I310d3d8deb489e097dd58ef8f8bc63aefe118051
Gerrit-Change-Number: 22103
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 24 Jan 2025 16:38:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-13520: Support in clause coercing
Posted by "Steve Carlin (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Fang-Yu Rao, Joe McDonnell, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22103
to look at the new patch set (#4).
Change subject: IMPALA-13520: Support in clause coercing
......................................................................
IMPALA-13520: Support in clause coercing
Calcite has special processing for any in clause. It has a callback
function that allows all the parameters to be coerced into its proper
type. While we do have a mechanism to do coercion, it only handles
functions, and the in clause is handled specially within Calcite.
So we use the Calcite mechanism to derive a common Impala type and
coerce all the parameters.
The CombineValueNodesRule is also needed for this change. There is a
test case in test_exprs.test where an in clause contains 10,000
params in side the IN clause (e.g. int_col IN (1, 2, 3, ..., 10000).
In this case, Calcite creates 10,000 Values RelNodes which takes way
too long to process on the execution side. The rule combines all
the Values RelNodes into one Values RelNode with many tuples, which
Impala handles quickly when converted into the physical Impala
PlanNode.
Change-Id: I310d3d8deb489e097dd58ef8f8bc63aefe118051
---
A java/calcite-planner/src/main/java/org/apache/impala/calcite/rules/CombineValueNodesRule.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteOptimizer.java
M java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteValidator.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeCoercionFactory.java
A java/calcite-planner/src/main/java/org/apache/impala/calcite/type/ImpalaTypeCoercionImpl.java
5 files changed, 260 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/22103/4
--
To view, visit http://gerrit.cloudera.org:8080/22103
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I310d3d8deb489e097dd58ef8f8bc63aefe118051
Gerrit-Change-Number: 22103
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-13520: Support in clause coercing
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22103 )
Change subject: IMPALA-13520: Support in clause coercing
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/18074/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22103
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I310d3d8deb489e097dd58ef8f8bc63aefe118051
Gerrit-Change-Number: 22103
Gerrit-PatchSet: 3
Gerrit-Owner: Steve Carlin <sc...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 24 Jan 2025 17:02:25 +0000
Gerrit-HasComments: No