You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Shant Hovsepian (Code Review)" <ge...@cloudera.org> on 2020/06/08 18:12:50 UTC
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Shant Hovsepian has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16007
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
IMPALA-8984: Uncorrelated scalar subqueries in the select list
Extend StmtRewriter with the ability to rewrite scalar subqueries in the
select list into cross joins. Currently the subquery must pass plan-time
checks to determine that it returns a single row which may miss cases
that may be valid at runtime or with more complex evaluation of the
predicate expressions in the planner. Support for correlated subqueries
will be a follow on change.
With this change Q9 of TPC-DS is supported, we now load the 'reasons'
table as part of the TPC-DS workload for use by Q9.
Testing:
* Added new analyzer tests, updated previous subquery tests
* test_queries.py::TestQueries::test_subquery
Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
---
M fe/src/main/java/org/apache/impala/analysis/SelectList.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-query/queries/QueryTest/subquery.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
A testdata/workloads/tpcds/queries/tpcds-q9.test
M tests/query_test/test_tpcds_queries.py
9 files changed, 515 insertions(+), 13 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/16007/3
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 3
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16007
to look at the new patch set (#6).
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
IMPALA-8984: Uncorrelated scalar subqueries in the select list
Extend StmtRewriter with the ability to rewrite scalar subqueries in the
select list into cross joins. Currently the subquery must pass plan-time
checks to determine that it returns a single row which may miss cases
that may be valid at runtime or with more complex evaluation of the
predicate expressions in the planner. Support for correlated subqueries
will be a follow on change.
Testing:
* Added new analyzer tests, updated previous subquery tests
* test_queries.py::TestQueries::test_subquery
* Added test_tpcds_q9 to e2e and planner tests
Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
---
M fe/src/main/java/org/apache/impala/analysis/SelectList.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-query/queries/QueryTest/subquery.test
M testdata/workloads/tpcds/queries/count.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
A testdata/workloads/tpcds/queries/tpcds-q9.test
M tests/query_test/test_tpcds_queries.py
M tests/util/parse_util.py
12 files changed, 1,438 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/16007/6
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 6
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6089/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 6
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2020 22:15:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Shant Hovsepian has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 6:
Fangu-Yu, FYI tweaked the reason table schema to be consistent with references to the column from other tables i.e. *_reason_sk
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 6
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2020 22:14:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6443/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 5
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sun, 28 Jun 2020 17:04:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 3:
(8 comments)
Thanks for the patch. A few comments below..overall it looks pretty good.
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@292
PS3, Line 292: "Invariant violated: Only subqueries that are gauranteed to return a "
nit: "guaranteed"
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@937
PS3, Line 937: * supported in FROM and WHERE clauses. The rewrite is performed in place and not in a
Update this comment for SELECT clause.
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1117
PS3, Line 1117: * Currently we only support very simple subqueries which return a single aggregate
Is this comment applicable for this patch ? Looks like this patch is supporting group-by columns in the subquery with a LIMIT 1 (based on one of your tests).
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1135
PS3, Line 1135: * In this case there is no aggregate function to gaurantee only a single row is
nit: spelling 'gaurantee'
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1137
PS3, Line 1137: * would be if the correlated predicates had primary primary key constraints.
nit: duplicate 'primary'
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1230
PS3, Line 1230: inlineView.setJoinOp(JoinOperator.CROSS_JOIN);
If the outer query has 2 tables joined in the FROM clause, then we add this inlineView with a CROSS_JOIN, is there any potential issue that may be uncovered ? I think if the outer tables are inner joined it should be ok. I am not completely sure if there is an outer join ..maybe worth adding a test.
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/functional-query/queries/QueryTest/subquery.test
File testdata/workloads/functional-query/queries/QueryTest/subquery.test:
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/functional-query/queries/QueryTest/subquery.test@1044
PS3, Line 1044: select id, 1+(select min(id) from functional.alltypessmall)
Would be useful to also add the Explain plan for some of these tests either here or as part of PlannerTest.
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
File testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test:
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test@3
PS3, Line 3: select case when (select count(*)
Cool that this query is now supported. Can we also add the plan for this under testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test ?
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 3
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 09 Jun 2020 06:48:43 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
IMPALA-8954: Uncorrelated scalar subqueries in the select list
Extend StmtRewriter with the ability to rewrite scalar subqueries in the
select list into cross joins. Currently the subquery must pass plan-time
checks to determine that it returns a single row which may miss cases
that may be valid at runtime or with more complex evaluation of the
predicate expressions in the planner. Support for correlated subqueries
will be a follow on change.
Testing:
* Added new analyzer tests, updated previous subquery tests
* test_queries.py::TestQueries::test_subquery
* Added test_tpcds_q9 to e2e and planner tests
Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Reviewed-on: http://gerrit.cloudera.org:8080/16007
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Tim Armstrong <ta...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/SelectList.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-query/queries/QueryTest/subquery.test
M testdata/workloads/tpcds/queries/count.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
A testdata/workloads/tpcds/queries/tpcds-q9.test
M tests/query_test/test_tpcds_queries.py
M tests/util/parse_util.py
13 files changed, 1,572 insertions(+), 18 deletions(-)
Approvals:
Tim Armstrong: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 10
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 7: Code-Review+1
(1 comment)
LGTM. One suggestion below for testing.
http://gerrit.cloudera.org:8080/#/c/16007/7/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test:
http://gerrit.cloudera.org:8080/#/c/16007/7/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test@9036
PS7, Line 9036: |--90:EXCHANGE [UNPARTITIONED]
This query seems to be a special case where both left and right inputs of the NLJ happen to be 1 row and hence none of the exchanges are BROADCAST which is the more common case. IT would be nice to have one explain with the common case (need not be the complex TPC-DS query..it could be one of the simplified queries).
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 7
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 06:10:12 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6066/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 4
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 27 Jun 2020 21:56:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Shant Hovsepian has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 8:
(11 comments)
Addressed some comments.
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1113
PS3, Line 1113:
> nit: implemented
Done
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1117
PS3, Line 1117: *
> Is this comment applicable for this patch ? Looks like this patch is suppor
Done
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1135
PS3, Line 1135: *
> nit: spelling 'gaurantee'
Done
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1230
PS3, Line 1230: // rewrite to a LOJ.
> If the outer query has 2 tables joined in the FROM clause, then we add this
Done
http://gerrit.cloudera.org:8080/#/c/16007/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/16007/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1114
PS5, Line 1114: implemeted
> nit: implemented
Done
http://gerrit.cloudera.org:8080/#/c/16007/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1126
PS5, Line 1126: flatten
> nit: flattened
Done
http://gerrit.cloudera.org:8080/#/c/16007/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1136
PS5, Line 1136: gaurantee
> nit: guarantee
Done
http://gerrit.cloudera.org:8080/#/c/16007/7/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test:
http://gerrit.cloudera.org:8080/#/c/16007/7/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test@9036
PS7, Line 9036: |--90:EXCHANGE [UNPARTITIONED]
> On a related note, we probably *should* be doing a broadcast when the right
Ack
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/functional-query/queries/QueryTest/subquery.test
File testdata/workloads/functional-query/queries/QueryTest/subquery.test:
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/functional-query/queries/QueryTest/subquery.test@1044
PS3, Line 1044: select id, 1+(select min(id) from functional.alltypessmall)
> Would be useful to also add the Explain plan for some of these tests either
Done
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/functional-query/queries/QueryTest/subquery.test@1086
PS3, Line 1086:
> Not sure what this means - without grouping?
Meant that the parent query had aggregation, and the outer and inner aggregations weren't interfering. This is a good case for subquery coalescing / merging down the line.
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
File testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test:
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test@3
PS3, Line 3: select case when (select count(*)
> Cool that this query is now supported. Can we also add the plan for this u
Done
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 8
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 19:33:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 9:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6492/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 9
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 20:04:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 9: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 9
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sun, 05 Jul 2020 22:03:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 9: Verified+1
Verifying since parent patch was verified
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 9
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sun, 05 Jul 2020 22:03:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6422/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 4
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jun 2020 20:01:10 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 8:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6490/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 8
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 14:07:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 6: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6089/
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 6
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 03:17:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 4: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6066/
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 4
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sun, 28 Jun 2020 03:07:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6487/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 6
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 03 Jul 2020 22:40:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 8: Code-Review+2
(6 comments)
I had a few nits about comments. I can +2 and merge after those are cleaned up. I was going to do that myself by I wasn't sure what one of them meant.
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1113
PS3, Line 1113:
nit: implemented
http://gerrit.cloudera.org:8080/#/c/16007/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/16007/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1114
PS5, Line 1114: implemeted
nit: implemented
http://gerrit.cloudera.org:8080/#/c/16007/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1126
PS5, Line 1126: flatten
nit: flattened
http://gerrit.cloudera.org:8080/#/c/16007/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1136
PS5, Line 1136: gaurantee
nit: guarantee
http://gerrit.cloudera.org:8080/#/c/16007/7/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test:
http://gerrit.cloudera.org:8080/#/c/16007/7/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test@9036
PS7, Line 9036: |--90:EXCHANGE [UNPARTITIONED]
> Good point added an end user sample query which illustrates this.
On a related note, we probably *should* be doing a broadcast when the right input is tiny regardless of the size of the input. There's no sense in setting up the extra fragment/exchange in that case. But I doubt this makes much of a difference.
Obviously we don't want to do this in this patch, and it probably isn't that important in general.
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/functional-query/queries/QueryTest/subquery.test
File testdata/workloads/functional-query/queries/QueryTest/subquery.test:
http://gerrit.cloudera.org:8080/#/c/16007/3/testdata/workloads/functional-query/queries/QueryTest/subquery.test@1086
PS3, Line 1086:
Not sure what this means - without grouping?
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 8
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 18:22:26 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16007
to look at the new patch set (#5).
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
IMPALA-8984: Uncorrelated scalar subqueries in the select list
Extend StmtRewriter with the ability to rewrite scalar subqueries in the
select list into cross joins. Currently the subquery must pass plan-time
checks to determine that it returns a single row which may miss cases
that may be valid at runtime or with more complex evaluation of the
predicate expressions in the planner. Support for correlated subqueries
will be a follow on change.
With this change Q9 of TPC-DS is supported, we now load the 'reasons'
table as part of the TPC-DS workload for use by Q9.
Testing:
* Added new analyzer tests, updated previous subquery tests
* test_queries.py::TestQueries::test_subquery
Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
---
M fe/src/main/java/org/apache/impala/analysis/SelectList.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-query/queries/QueryTest/subquery.test
M testdata/workloads/tpcds/queries/count.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
A testdata/workloads/tpcds/queries/tpcds-q9.test
M tests/query_test/test_tpcds_queries.py
M tests/util/parse_util.py
12 files changed, 1,456 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/16007/5
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 5
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Fang-Yu Rao, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16007
to look at the new patch set (#7).
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
IMPALA-8984: Uncorrelated scalar subqueries in the select list
Extend StmtRewriter with the ability to rewrite scalar subqueries in the
select list into cross joins. Currently the subquery must pass plan-time
checks to determine that it returns a single row which may miss cases
that may be valid at runtime or with more complex evaluation of the
predicate expressions in the planner. Support for correlated subqueries
will be a follow on change.
Testing:
* Added new analyzer tests, updated previous subquery tests
* test_queries.py::TestQueries::test_subquery
* Added test_tpcds_q9 to e2e and planner tests
Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
---
M fe/src/main/java/org/apache/impala/analysis/SelectList.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-query/queries/QueryTest/subquery.test
M testdata/workloads/tpcds/queries/count.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
A testdata/workloads/tpcds/queries/tpcds-q9.test
M tests/query_test/test_tpcds_queries.py
M tests/util/parse_util.py
12 files changed, 1,438 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/16007/7
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 7
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5963/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 3
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2020 18:16:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Shant Hovsepian has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 8:
(1 comment)
> Patch Set 7: Code-Review+1
>
> (1 comment)
>
> LGTM. One suggestion below for testing.
http://gerrit.cloudera.org:8080/#/c/16007/7/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
File testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test:
http://gerrit.cloudera.org:8080/#/c/16007/7/testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test@9036
PS7, Line 9036: |--90:EXCHANGE [UNPARTITIONED]
> This query seems to be a special case where both left and right inputs of t
Good point added an end user sample query which illustrates this.
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 8
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 13:40:37 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 7
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 08:58:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 3: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5963/
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 3
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2020 23:09:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6090/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 7
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 03:58:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Shant Hovsepian has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 4:
(9 comments)
Some more tests and review comments addressed. Still want to get a good test run out of jenkins.
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@292
PS3, Line 292: "Invariant violated: Only subqueries that are guaranteed to return a "
> nit: "guaranteed"
Done
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@937
PS3, Line 937: * supported in the FROM clause, WHERE clause and SELECT list. The rewrite is
> Update this comment for SELECT clause.
Done
http://gerrit.cloudera.org:8080/#/c/16007/3/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1137
PS3, Line 1137: * returned per group so a run time cardinality check must be applied. An exception
> nit: duplicate 'primary'
Done
http://gerrit.cloudera.org:8080/#/c/16007/4/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/16007/4/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1117
PS4, Line 1117: *
Done
http://gerrit.cloudera.org:8080/#/c/16007/4/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1117
PS4, Line 1117: *
Done
http://gerrit.cloudera.org:8080/#/c/16007/4/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1135
PS4, Line 1135: *
Done
http://gerrit.cloudera.org:8080/#/c/16007/4/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@1230
PS4, Line 1230: // rewrite to a LOJ.
I added a test for this. I know it feels weird but since the slotref for the subquery is marked as materialized and the other join queries get bound by the USING/ON clause, nothing explodes. Since there are scalar subqueries, the only weird situation is if the cardinality of all the joins where equal to 1 then it might get reordered but the results would still be correct.
If it were a correlated subquery then we've need to handle things more carefully, but that's for a later commit.
http://gerrit.cloudera.org:8080/#/c/16007/4/testdata/workloads/functional-query/queries/QueryTest/subquery.test
File testdata/workloads/functional-query/queries/QueryTest/subquery.test:
http://gerrit.cloudera.org:8080/#/c/16007/4/testdata/workloads/functional-query/queries/QueryTest/subquery.test@1044
PS4, Line 1044: select id, 1+(select min(id) from functional.alltypessmall)
Added the tpc-ds query, it's a pretty complex plan.
http://gerrit.cloudera.org:8080/#/c/16007/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
File testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test:
http://gerrit.cloudera.org:8080/#/c/16007/4/testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test@3
PS4, Line 3: select case when (select count(*)
Done
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 4
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 25 Jun 2020 19:38:11 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6243/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 3
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 Jun 2020 18:58:11 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16007 )
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6488/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 7
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 04 Jul 2020 04:25:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8984: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16007
to look at the new patch set (#4).
Change subject: IMPALA-8984: Uncorrelated scalar subqueries in the select list
......................................................................
IMPALA-8984: Uncorrelated scalar subqueries in the select list
Extend StmtRewriter with the ability to rewrite scalar subqueries in the
select list into cross joins. Currently the subquery must pass plan-time
checks to determine that it returns a single row which may miss cases
that may be valid at runtime or with more complex evaluation of the
predicate expressions in the planner. Support for correlated subqueries
will be a follow on change.
With this change Q9 of TPC-DS is supported, we now load the 'reasons'
table as part of the TPC-DS workload for use by Q9.
Testing:
* Added new analyzer tests, updated previous subquery tests
* test_queries.py::TestQueries::test_subquery
Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
---
M fe/src/main/java/org/apache/impala/analysis/SelectList.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-query/queries/QueryTest/subquery.test
M testdata/workloads/tpcds/queries/count.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
A testdata/workloads/tpcds/queries/tpcds-q9.test
M tests/query_test/test_tpcds_queries.py
11 files changed, 1,455 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/16007/4
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 4
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Fang-Yu Rao, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16007
to look at the new patch set (#9).
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
IMPALA-8954: Uncorrelated scalar subqueries in the select list
Extend StmtRewriter with the ability to rewrite scalar subqueries in the
select list into cross joins. Currently the subquery must pass plan-time
checks to determine that it returns a single row which may miss cases
that may be valid at runtime or with more complex evaluation of the
predicate expressions in the planner. Support for correlated subqueries
will be a follow on change.
Testing:
* Added new analyzer tests, updated previous subquery tests
* test_queries.py::TestQueries::test_subquery
* Added test_tpcds_q9 to e2e and planner tests
Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
---
M fe/src/main/java/org/apache/impala/analysis/SelectList.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-query/queries/QueryTest/subquery.test
M testdata/workloads/tpcds/queries/count.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
A testdata/workloads/tpcds/queries/tpcds-q9.test
M tests/query_test/test_tpcds_queries.py
M tests/util/parse_util.py
13 files changed, 1,572 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/16007/9
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 9
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8954: Uncorrelated scalar subqueries in the select list
Posted by "Shant Hovsepian (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Fang-Yu Rao, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16007
to look at the new patch set (#8).
Change subject: IMPALA-8954: Uncorrelated scalar subqueries in the select list
......................................................................
IMPALA-8954: Uncorrelated scalar subqueries in the select list
Extend StmtRewriter with the ability to rewrite scalar subqueries in the
select list into cross joins. Currently the subquery must pass plan-time
checks to determine that it returns a single row which may miss cases
that may be valid at runtime or with more complex evaluation of the
predicate expressions in the planner. Support for correlated subqueries
will be a follow on change.
Testing:
* Added new analyzer tests, updated previous subquery tests
* test_queries.py::TestQueries::test_subquery
* Added test_tpcds_q9 to e2e and planner tests
Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
---
M fe/src/main/java/org/apache/impala/analysis/SelectList.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
M testdata/datasets/tpcds/tpcds_schema_template.sql
M testdata/workloads/functional-planner/queries/PlannerTest/subquery-rewrite.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpcds-all.test
M testdata/workloads/functional-query/queries/QueryTest/subquery.test
M testdata/workloads/tpcds/queries/count.test
A testdata/workloads/tpcds/queries/tpcds-decimal_v2-q9.test
A testdata/workloads/tpcds/queries/tpcds-q9.test
M tests/query_test/test_tpcds_queries.py
M tests/util/parse_util.py
13 files changed, 1,572 insertions(+), 18 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/16007/8
--
To view, visit http://gerrit.cloudera.org:8080/16007
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibcf55d26889aa01d69bb85f18c9241dda095fb66
Gerrit-Change-Number: 16007
Gerrit-PatchSet: 8
Gerrit-Owner: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fa...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Shant Hovsepian <sh...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>