You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Daniel Becker (Code Review)" <ge...@cloudera.org> on 2023/12/06 12:48:21 UTC
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Daniel Becker has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20752
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
IMPALA-12001: Informative error message for complex types with DISTINCT
Before this change, queries with SELECT DISTINCT on a complex type
failed.
With structs, we got a FE exception:
use functional_parquet;
select distinct(struct_val) from alltypes_structs;
ERROR: IllegalStateException: null
With collections the BE hits a DCHECK and crashes:
use functional_parquet;
select distinct(arr1) from complextypes_arrays;
Socket error 104: [Errno 104] Connection reset by peer
Aggregate functions with complex DISTINCT parameters also failed without
a clear error message. For example:
select count(distinct struct_val) from alltypes_structs;
select count(distinct arr1) from complextypes_arrays;
To support DISTINCT for complex types we would need to implement
equality and hash for them. We are not planning to do it in the near
future, so this change introduces informative error messages in these
cases.
Testing:
- added test queries for SELECT DISTINCT and SELECT COUNT(DISTINCT ...)
with arrays, maps and structs, expecting the correct error messages.
Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
---
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
M testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test
5 files changed, 53 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/20752/2
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
IMPALA-12001: Informative error message for complex types with DISTINCT
Before this change, queries with SELECT DISTINCT on a complex type
failed.
With structs, we got a FE exception:
use functional_parquet;
select distinct(struct_val) from alltypes_structs;
ERROR: IllegalStateException: null
With collections, the BE hits a DCHECK and crashes:
use functional_parquet;
select distinct(arr1) from complextypes_arrays;
Socket error 104: [Errno 104] Connection reset by peer
Aggregate functions with complex DISTINCT parameters also failed without
a clear error message. For example:
select count(distinct struct_val) from alltypes_structs;
select count(distinct arr1) from complextypes_arrays;
To support DISTINCT for complex types we would need to implement
equality and hash for them. We are not planning to do it in the near
future, so this change introduces informative error messages in these
cases.
Testing:
- added test queries for SELECT DISTINCT and SELECT COUNT(DISTINCT ...)
with arrays, maps and structs, expecting the correct error messages.
Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Reviewed-on: http://gerrit.cloudera.org:8080/20752
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
M testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test
5 files changed, 58 insertions(+), 7 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 5
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14596/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Dec 2023 13:15:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
Patch Set 2:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/20752/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/20752/2//COMMIT_MSG@18
PS2, Line 18:
nit: comma
http://gerrit.cloudera.org:8080/#/c/20752/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/20752/2/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@565
PS2, Line 565: if (selectList_.isDistinct() && expr.getType().isComplexType()) {
Just a question: there are numerous cases when we use format strings as error messages instead of simple concatenation. Is there a guideline for determining when to use format strings as opposed to concatenation?
http://gerrit.cloudera.org:8080/#/c/20752/2/testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
File testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test:
http://gerrit.cloudera.org:8080/#/c/20752/2/testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test@456
PS2, Line 456: =
nit: one superfluous '='
http://gerrit.cloudera.org:8080/#/c/20752/2/testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
File testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test:
http://gerrit.cloudera.org:8080/#/c/20752/2/testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test@437
PS2, Line 437: =====
nit: one superfluous '='
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Dec 2023 15:54:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Daniel Becker (Code Review)" <ge...@cloudera.org>.
Daniel Becker has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
IMPALA-12001: Informative error message for complex types with DISTINCT
Before this change, queries with SELECT DISTINCT on a complex type
failed.
With structs, we got a FE exception:
use functional_parquet;
select distinct(struct_val) from alltypes_structs;
ERROR: IllegalStateException: null
With collections, the BE hits a DCHECK and crashes:
use functional_parquet;
select distinct(arr1) from complextypes_arrays;
Socket error 104: [Errno 104] Connection reset by peer
Aggregate functions with complex DISTINCT parameters also failed without
a clear error message. For example:
select count(distinct struct_val) from alltypes_structs;
select count(distinct arr1) from complextypes_arrays;
To support DISTINCT for complex types we would need to implement
equality and hash for them. We are not planning to do it in the near
future, so this change introduces informative error messages in these
cases.
Testing:
- added test queries for SELECT DISTINCT and SELECT COUNT(DISTINCT ...)
with arrays, maps and structs, expecting the correct error messages.
Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
---
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M testdata/workloads/functional-query/queries/QueryTest/nested-array-in-select-list.test
M testdata/workloads/functional-query/queries/QueryTest/nested-map-in-select-list.test
M testdata/workloads/functional-query/queries/QueryTest/nested-struct-in-select-list.test
5 files changed, 58 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/20752/3
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/14638/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 10:49:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
Patch Set 2: Code-Review+1
lgtm apart from comments by Peter
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 2
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Wed, 06 Dec 2023 17:47:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 14:07:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 18:38:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 14:06:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9997/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 4
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 14:07:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12001: Informative error message for complex types with DISTINCT
Posted by "Peter Rozsa (Code Review)" <ge...@cloudera.org>.
Peter Rozsa has posted comments on this change. ( http://gerrit.cloudera.org:8080/20752 )
Change subject: IMPALA-12001: Informative error message for complex types with DISTINCT
......................................................................
Patch Set 3: Code-Review+1
Thank you, Daniel!
--
To view, visit http://gerrit.cloudera.org:8080/20752
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibe2642d1683a10fd05a95e2ad8470d16f0d5242c
Gerrit-Change-Number: 20752
Gerrit-PatchSet: 3
Gerrit-Owner: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Daniel Becker <da...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Noemi Pap-Takacs <np...@cloudera.com>
Gerrit-Reviewer: Peter Rozsa <pr...@cloudera.com>
Gerrit-Comment-Date: Mon, 11 Dec 2023 10:45:00 +0000
Gerrit-HasComments: No