You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "liuyao (Code Review)" <ge...@cloudera.org> on 2021/04/27 04:05:01 UTC
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
liuyao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17344
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
IMPALA-10677: Set selectivity of Not-equal
Calculate binary predicate selectivity if one of the children is
a slotref and the other children are all constant.
eg. something like "col = 5", but not "2 * col = 10"
selectivity = 1 - 1/ndv
Testing:
Modify the function testNeSelectivity() of the
ExprCardinalityTest.java, change -1 to the correct value.
Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
12 files changed, 60 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/17344/1
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Qifan Chen, Zoltan Borok-Nagy, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17344
to look at the new patch set (#6).
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
IMPALA-7560: Set selectivity of Not-equal
Calculate binary predicate selectivity if one of the children is
a slotref and the other children are all constant.
eg. something like "col != 5", but not "2 * col != 10"
selectivity = 1 - 1/ndv
Testing:
Modify the function testNeSelectivity() of the
ExprCardinalityTest.java, change -1 to the correct value.
Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
13 files changed, 109 insertions(+), 100 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/17344/6
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 2: Code-Review+1
Looks good!
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 14:24:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@244
PS1, Line 244:
> This condition can be checked at the beginning together with !isSingleColum
Done
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@249
PS1, Line 249: to selectivity is 0.
> Repetition in both then and else branch.
Done
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java:
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@316
PS1, Line 316: testNeSelectivity
> May add a couple test cases for an empty table.
Done
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 11:46:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7108/
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 17:36:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@257
PS3, Line 257: distinctValues
> I wonder if distinctValues includes the null. If so, we may need to compute
If the column contains null, but not all null, then distinctValues does not contain null
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 04 Jun 2021 03:40:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 4:
(7 comments)
Thanks you for working on this. Seems like NULL values are not handled correctly right now.
I think we should either use the number of NULLs from column stats and return correct results, or we should not set selectivity when the parameter is NULL.
Please note that the NOT DISTINCT FROM test also produces wrong values in case of NULLs.
http://gerrit.cloudera.org:8080/#/c/17344/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17344/4//COMMIT_MSG@7
PS4, Line 7: IMPALA-10677
You might want to use IMPALA-7560 as this was the original Jira and it contains more information.
http://gerrit.cloudera.org:8080/#/c/17344/4//COMMIT_MSG@11
PS4, Line 11: col = 5", but not "2 * col = 10
nit: wouldn't be better to use != in the example?
"col != 5", but not "2 * col != 10"
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@257
PS3, Line 257: distinctValues
> Okay. Thanks for the verification. We could make an adjustment here if ther
SlotRef retrieves the NDV from the SlotDescriptor's ColumnStats in SlotRef.adjustNumDistinctValues() .
ColumnStats also has NumNulls which could be used here:
https://github.com/apache/impala/blob/954eb5c85d329af7690698cdc4d0f409260e6d18/fe/src/main/java/org/apache/impala/catalog/ColumnStats.java#L221
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java:
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@a372
PS4, Line 372:
That seems wrong looking at the data. I think it should be 1.
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@344
PS4, Line 344: 7300
NULL values are not included in the NDV, also null count is stored separately in the column stats.
I think we should handle NULLs specially, as mentioned in the above comment.
I.e. we should have the same values here that are commented out in the original test case.In this case 1.0, as there are no NULL values for 'id' in the alltypes table.
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@355
PS4, Line 355: 1 - 1.0/1
I think the expected value is 0 here, as null_str is all nulls.
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@359
PS4, Line 359: some_nulls is not distinct from 'foo'",
I think it'd be worth to add a test case with
some_nulls is distinct from null
I think the expected value would be "6.0 / 26.0", i.e. (NumRows - NumNulls) / (NumRows)
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 11 Jun 2021 15:36:03 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 2:
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/7109/
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 17:39:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7208/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 09 Jun 2021 06:49:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 29 Apr 2021 13:17:15 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/17344/6/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/6/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@256
PS6, Line 256: selectivity_ = 1.0 / distinctValues;
> I think if we have null stats, the selectivity for EQ case should be comput
Maybe I submit another patch to modify the selectivity of EQ to be better. This patch only modifies the selectivity of NE
http://gerrit.cloudera.org:8080/#/c/17344/6/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@258
PS6, Line 258: //
> nit. May add a comment here: For case <column> IS DISTINCT FROM NULL
Done
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 22 Jun 2021 02:44:20 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Qifan Chen, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17344
to look at the new patch set (#4).
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
IMPALA-10677: Set selectivity of Not-equal
Calculate binary predicate selectivity if one of the children is
a slotref and the other children are all constant.
eg. something like "col = 5", but not "2 * col = 10"
selectivity = 1 - 1/ndv
Testing:
Modify the function testNeSelectivity() of the
ExprCardinalityTest.java, change -1 to the correct value.
Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
13 files changed, 91 insertions(+), 97 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/17344/4
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Qifan Chen, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17344
to look at the new patch set (#3).
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
IMPALA-10677: Set selectivity of Not-equal
Calculate binary predicate selectivity if one of the children is
a slotref and the other children are all constant.
eg. something like "col = 5", but not "2 * col = 10"
selectivity = 1 - 1/ndv
Testing:
Modify the function testNeSelectivity() of the
ExprCardinalityTest.java, change -1 to the correct value.
Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
13 files changed, 91 insertions(+), 93 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/17344/3
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8657/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 12:04:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 6:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/17344/4//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17344/4//COMMIT_MSG@7
PS4, Line 7: IMPALA-7560:
> You might want to use IMPALA-7560 as this was the original Jira and it cont
Done
http://gerrit.cloudera.org:8080/#/c/17344/4//COMMIT_MSG@11
PS4, Line 11: col != 5", but not "2 * col !=
> nit: wouldn't be better to use != in the example?
Done
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@257
PS3, Line 257: || op_ == Ope
> SlotRef retrieves the NDV from the SlotDescriptor's ColumnStats in SlotRef.
Done
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java:
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@a372
PS4, Line 372:
> That seems wrong looking at the data. I think it should be 1.
Done
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@344
PS4, Line 344:
> NULL values are not included in the NDV, also null count is stored separate
Done
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@355
PS4, Line 355: 0);
> I think the expected value is 0 here, as null_str is all nulls.
Done
http://gerrit.cloudera.org:8080/#/c/17344/4/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@359
PS4, Line 359: some_nulls is not distinct from 'foo'",
> I think it'd be worth to add a test case with
Done
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 18 Jun 2021 07:30:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 4: Code-Review+1
(1 comment)
Thanks again for the verification. Looks good!
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@257
PS3, Line 257: distinctValues
> If the column contains null, but not all null, then distinctValues does not
Okay. Thanks for the verification. We could make an adjustment here if there is a way to detect this situation. But it looks like Expr does not tell us # of nulls in general.
Done.
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 04 Jun 2021 13:13:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 09 Jun 2021 12:32:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 6: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 21 Jun 2021 08:12:18 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 7: Code-Review+1
(1 comment)
Thanks a lot for the work which looks great!
http://gerrit.cloudera.org:8080/#/c/17344/6/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/6/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@256
PS6, Line 256: selectivity_ = 1.0 / distinctValues;
> Maybe I submit another patch to modify the selectivity of EQ to be better.
Done
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 22 Jun 2021 14:05:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7102/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Apr 2021 04:06:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7108/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Wed, 28 Apr 2021 11:45:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Apr 2021 09:56:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Qifan Chen, Zoltan Borok-Nagy, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17344
to look at the new patch set (#5).
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
IMPALA-7560: Set selectivity of Not-equal
Calculate binary predicate selectivity if one of the children is
a slotref and the other children are all constant.
eg. something like "col != 5", but not "2 * col != 10"
selectivity = 1 - 1/ndv
Testing:
Modify the function testNeSelectivity() of the
ExprCardinalityTest.java, change -1 to the correct value.
Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
13 files changed, 108 insertions(+), 100 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/17344/5
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 5
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8843/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 04 Jun 2021 03:57:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8660/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 29 Apr 2021 07:32:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 7:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8964/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Tue, 22 Jun 2021 03:01:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7240/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 21 Jun 2021 02:17:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 1:
(3 comments)
Looks good!
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@244
PS1, Line 244: distinctValues > 0
This condition can be checked at the beginning together with !isSingleColumnPredicate().
If distinctValues == 0, should we set the selectivity_ to 0 for both EQ and NE case?
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@249
PS1, Line 249: selectivity_ = Math.max(0, Math.min(1, selectivity_));
Repetition in both then and else branch.
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java:
http://gerrit.cloudera.org:8080/#/c/17344/1/fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java@316
PS1, Line 316: testNeSelectivity
May add a couple test cases for an empty table.
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Apr 2021 17:33:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7109/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 28 Apr 2021 11:47:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 7: Code-Review+2
Great work, thank you!
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 23 Jun 2021 14:05:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Qifan Chen, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17344
to look at the new patch set (#2).
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
IMPALA-10677: Set selectivity of Not-equal
Calculate binary predicate selectivity if one of the children is
a slotref and the other children are all constant.
eg. something like "col = 5", but not "2 * col = 10"
selectivity = 1 - 1/ndv
Testing:
Modify the function testNeSelectivity() of the
ExprCardinalityTest.java, change -1 to the correct value.
Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
12 files changed, 89 insertions(+), 91 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/17344/2
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 2
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 6:
(2 comments)
Looks very good to me!
http://gerrit.cloudera.org:8080/#/c/17344/6/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/6/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@256
PS6, Line 256: selectivity_ = 1.0 / distinctValues;
I think if we have null stats, the selectivity for EQ case should be computed as follows.
Let N = num nulls, and M = num of not nulls.
Selectivity = (M/distinctValues) / (M + N)
http://gerrit.cloudera.org:8080/#/c/17344/6/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@258
PS6, Line 258: if
nit. May add a comment here: For case <column> IS DISTINCT FROM NULL
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 21 Jun 2021 17:40:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7253/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 23 Jun 2021 06:58:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8950/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 18 Jun 2021 07:44:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 6: Code-Review+1
Thanks for applying the changes. LGTM!
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 6
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Mon, 21 Jun 2021 15:06:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Wed, 23 Jun 2021 12:49:19 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
IMPALA-7560: Set selectivity of Not-equal
Calculate binary predicate selectivity if one of the children is
a slotref and the other children are all constant.
eg. something like "col != 5", but not "2 * col != 10"
selectivity = 1 - 1/ndv
Testing:
Modify the function testNeSelectivity() of the
ExprCardinalityTest.java, change -1 to the correct value.
Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Reviewed-on: http://gerrit.cloudera.org:8080/17344
Reviewed-by: Qifan Chen <qc...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Zoltan Borok-Nagy <bo...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
13 files changed, 110 insertions(+), 100 deletions(-)
Approvals:
Qifan Chen: Looks good to me, but someone else must approve
Impala Public Jenkins: Verified
Zoltan Borok-Nagy: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 8
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8644/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 1
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 27 Apr 2021 04:25:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Qifan Chen, Zoltan Borok-Nagy, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17344
to look at the new patch set (#7).
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
IMPALA-7560: Set selectivity of Not-equal
Calculate binary predicate selectivity if one of the children is
a slotref and the other children are all constant.
eg. something like "col != 5", but not "2 * col != 10"
selectivity = 1 - 1/ndv
Testing:
Modify the function testNeSelectivity() of the
ExprCardinalityTest.java, change -1 to the correct value.
Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/analytic-fns.test
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
13 files changed, 110 insertions(+), 100 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/17344/7
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 7
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8949/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 5
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 18 Jun 2021 07:21:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "liuyao (Code Review)" <ge...@cloudera.org>.
liuyao has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 4:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@257
PS3, Line 257: distinctValues
> I wonder if distinctValues includes the null. If so, we may need to compute
If all values of the column are null, the distinctValues is 1; if not, the distinctValues does not includes null
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
File fe/src/test/java/org/apache/impala/planner/CardinalityTest.java:
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@70
PS3, Line 70: verifyCardinality(
: "SELECT id FROM functional.alltypes WHE
> Should we remove this comment, since the calculation is correct with the pa
Done
http://gerrit.cloudera.org:8080/#/c/17344/3/testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
File testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test:
http://gerrit.cloudera.org:8080/#/c/17344/3/testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test@160
PS3, Line 160: from functional.alltypestiny
> Remove?
Done
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 4
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 04 Jun 2021 03:37:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/7113/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 29 Apr 2021 07:13:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10677: Set selectivity of Not-equal
Posted by "Qifan Chen (Code Review)" <ge...@cloudera.org>.
Qifan Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-10677: Set selectivity of Not-equal
......................................................................
Patch Set 3: Code-Review+1
(3 comments)
Looks great!
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@257
PS3, Line 257: distinctValues
I wonder if distinctValues includes the null. If so, we may need to compute 1 - 1 / (distinctValues-1) here.
Can you please double check?
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
File fe/src/test/java/org/apache/impala/planner/CardinalityTest.java:
http://gerrit.cloudera.org:8080/#/c/17344/3/fe/src/test/java/org/apache/impala/planner/CardinalityTest.java@70
PS3, Line 70: // Assume classic 0.1 selectivity for other operators
: // IMPALA-7560 says this should be revised.
Should we remove this comment, since the calculation is correct with the patch?
http://gerrit.cloudera.org:8080/#/c/17344/3/testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
File testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test:
http://gerrit.cloudera.org:8080/#/c/17344/3/testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test@160
PS3, Line 160: # Bug, expected cardinality ~7
Remove?
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 3
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Thu, 03 Jun 2021 18:25:42 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7560: Set selectivity of Not-equal
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17344 )
Change subject: IMPALA-7560: Set selectivity of Not-equal
......................................................................
Patch Set 5:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17344/5/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/17344/5/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@268
PS5, Line 268: selectivity_ *= (double) (numRows - slotDesc.getStats().getNumNulls()) / numRows;
line too long (91 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/17344
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icd6f5945840ea2a8194d72aa440ddfa6915cbb3a
Gerrit-Change-Number: 17344
Gerrit-PatchSet: 5
Gerrit-Owner: liuyao <li...@sensorsdata.cn>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: liuyao <li...@sensorsdata.cn>
Gerrit-Comment-Date: Fri, 18 Jun 2021 07:02:30 +0000
Gerrit-HasComments: Yes