You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2022/05/18 16:06:23 UTC
[Impala-ASF-CR] WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18543
Change subject: WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
......................................................................
WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
The original selectivity of 1.0/ndv and 1.0 - 1.0/ndv make sense
for large NDVs, but the result is 1.0 and 0.0 in case of ndv==1,
which can lead to extremely wrong cardinality estimates.
Adding 1 to the ndv in the formulas also accounts for the possibility
of 0 matching elements in the dataset and helps avoiding extreme
selectivity. IMPALA-7601 contains some discussion about these formulas.
The change is in WIP state because I had not time to run the
tests yet.
Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18543/1
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10600/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 15:01:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 4:
Build Failed
https://jenkins.impala.io/job/gerrit-code-review-checks/10602/ : Initial code review checks failed. See linked job for details on the failure.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 17:43:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme = and != selectivity for NDV=1
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme = and != selectivity for NDV=1
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@267
PS1, Line 267: if (distinctValues == 1) distinctValues = 2;
> Yeah the NOT col = 1 is a good use case to justify this. I am good with i
Hmm..not sure what I meant by the last sentence in my comment above. Pls ignore that part. Looking at the Jenkins test output, I see several diffs in the hash join cardinality estimates for TPC-DS queries. I was worried about plan changes but wasn't expecting it for TPC-DS since I doubt there are columns with ndv=1. However, what might be happening is that the ndv gets adjusted based on filtering that was applied earlier in the plan. A join's cardinality estimate formula leverages the ndv coming from the child which may have been adjusted (reduced).
We could look at all the plan diffs but it would need some time to validate functionally and with a performance run. Alternatively, we could limit the scope of this fix to the != predicates.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 02:26:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11301: Fix extreme = and != selectivity for NDV=1
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme = and != selectivity for NDV=1
......................................................................
Patch Set 2:
Found out the reasons behind the failed tests:
- analytic functions return ndv 1 (incorrectly)
- "having" clause returns ndv 1 (incorrectly)
- several columns in tpcds (scale factor 1) have ndv=1 - in these cases the = predicate leading selectivity of 1.0 seems optimal, as they always filter for the value that exists
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 10:33:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme = and != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme = and != selectivity for NDV=1
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10594/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 May 2022 20:33:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme = and != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme = and != selectivity for NDV=1
......................................................................
Patch Set 2: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/8117/
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 00:57:34 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10593/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 May 2022 16:26:38 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
IMPALA-11301: Fix extreme != selectivity for NDV=1
The original selectivity of 1.0 - 1.0/ndv makes sense for
large NDVs, but the result is 0.0 in case of ndv==1, which
leads to a cardinality of 1 even for huge tables. The new
selectivity is 0.5.
Note that as the formula for = is not changed (1.0/ndv),
NOT col="const" will still lead to 0.0 selectivity if ndv=1.
Changing the formula of NOT or = would have caused a lot of
subtle changes in plans in tests, so I don't want to touch
those before coming to wider agreement about the correct
approach.
IMPALA-7601 contains some discussion about these formulas.
Testing:
- added a regression test
Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Reviewed-on: http://gerrit.cloudera.org:8080/18543
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
2 files changed, 12 insertions(+), 1 deletion(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 6
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 May 2022 14:01:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@267
PS1, Line 267: selectivity_ = 1.0 / (distinctValues + 1);
> Hmm..not sure what I meant by the last sentence in my comment above. Pls i
Chosen to only touch the != case in the end to avoid side effects. I became unsure about the correct solution - I think that we should try to avoid extreme small selectivities in general, for example we could assume that 1% of rows always survives the predicate (unless we can really rule this out, for example when ndv=row_count) to rule out that a much bigger table falling on the build side. Generally it is not a huge issue if mistakenly the build side will have more rows than the probe side as long as it fits to memory, but should really try to avoid extremely large build sides.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 14:57:10 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 17:57:55 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 19:24:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 3: Code-Review+1
Latest changes lgtm. Only other thought was about the unit test itself. It relies on the current behavior of row_number ndv estimate. Were you thinking of adding any other test against a table whose column had ndv=1 ?
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 16:31:00 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@267
PS1, Line 267: selectivity_ = 1.0 / (distinctValues + 1);
If NDV == 1, this would now produce 0.5 instead of 1 which is not right. I suspect this change will change a lot of plans especially for predicates on boolean columns (ndv = 2). IMHO It would be better to restrict the fix to the non-equality predicates.
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@273
PS1, Line 273: selectivity_ = 1.0 - 1.0 / (distinctValues + 1);
Since the distinctValues == 1 is a special case which causes 0 to be produced with the old formula, could we just explicitly check for distinctValues == 1 and only in that case use the new formula ? For the other values, old formula should still be ok right ? Just want to see if a minimal change can be done since formula changes have to go through much more testing.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 May 2022 17:21:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11301: Fix extreme = and != selectivity for NDV=1
Posted by "Aman Sinha (Code Review)" <ge...@cloudera.org>.
Aman Sinha has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme = and != selectivity for NDV=1
......................................................................
Patch Set 2: Code-Review+2
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@267
PS1, Line 267: if (distinctValues == 1) distinctValues = 2;
> Yes, and I do think that 0.5 is "more right" than 1.
Yeah the NOT col = 1 is a good use case to justify this. I am good with it. In any case the cardinality will be rounded to the next integer value.
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@273
PS1, Line 273: if (op_ == Operator.DISTINCT_FROM && rChildIsNull) {
> I have changed the patch to only affect ndv=1.
Thanks for making the change to do an explicit check. The presence or absence of a value is not possible to be handled through NDV alone. If we had Histogram or Most Common Values, those keep track of actual values which would help.
Marking this resolved.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 May 2022 22:37:51 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11301: Fix extreme = and != selectivity for NDV=1
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme = and != selectivity for NDV=1
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@267
PS1, Line 267: if (distinctValues == 1) distinctValues = 2;
> If NDV == 1, this would now produce 0.5 instead of 1 which is not right.
Yes, and I do think that 0.5 is "more right" than 1.
For example WHERE NOT col = 1; would still lead to selectivity of 0.0 as NOT is calculated as selectivity = 1 -child_selectivity.
The patch was changed to only affect ndv=1, but it still affect both = and !=
http://gerrit.cloudera.org:8080/#/c/18543/1/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@273
PS1, Line 273: if (op_ == Operator.DISTINCT_FROM && rChildIsNull) {
> Since the distinctValues == 1 is a special case which causes 0 to be produc
I have changed the patch to only affect ndv=1.
The reason is the large number of tests it broke and the complexity of handling the case when distinctValues + 1 becomes larger than than the maximum ndv for a give type (for example 2 for booleans).
I am not sure whether "distinctValues + 1" "distinctValues" is more correct. as the original formula doesn't count with the possibility of the value not being present in the table.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 May 2022 20:23:28 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-11301: Fix extreme = and != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme = and != selectivity for NDV=1
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8117/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 May 2022 20:24:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8118/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 14:57:27 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
Posted by "Abhishek Rawat (Code Review)" <ge...@cloudera.org>.
Abhishek Rawat has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 May 2022 16:40:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
Posted by "Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>.
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: WIP: IMPALA-11301: Fix = and != selectivity for very low NDVs
......................................................................
Patch Set 1: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Wed, 18 May 2022 16:57:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme = and != selectivity for NDV=1
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Gabor Kaszab, Abhishek Rawat, Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18543
to look at the new patch set (#2).
Change subject: IMPALA-11301: Fix extreme = and != selectivity for NDV=1
......................................................................
IMPALA-11301: Fix extreme = and != selectivity for NDV=1
The original selectivity of 1.0/ndv and 1.0 - 1.0/ndv make sense
for large NDVs, but the result is 1.0 and 0.0 in case of ndv==1,
which can lead to extremely wrong cardinality estimates.
IMPALA-7601 contains some discussion about these formulas.
Testing:
- added a regression test
Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
2 files changed, 11 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18543/2
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/10603/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 18:10:21 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Gabor Kaszab, Abhishek Rawat, Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18543
to look at the new patch set (#3).
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
IMPALA-11301: Fix extreme != selectivity for NDV=1
The original selectivity of 1.0 - 1.0/ndv makes sense for
large NDVs, but the result is 0.0 in case of ndv==1, which
leads to a cardinality of 1 even for huge tables. The new
selectivity is 0.5.
Note that as the formula for = is not changed (1.0/ndv),
NOT col="const" will still lead to 0.0 selectivity if ndv=1.
Changing the formula of NOT or = would have caused a lot of
subtle changes in plans in tests, so I don't want to touch
those before coming to wider agreement about the correct
approach.
IMPALA-7601 contains some discussion about these formulas.
Testing:
- added a regression test
Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
2 files changed, 9 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18543/3
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8122/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 May 2022 09:35:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 5
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Fri, 20 May 2022 09:35:57 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Aman Sinha, Gabor Kaszab, Abhishek Rawat, Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18543
to look at the new patch set (#4).
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
IMPALA-11301: Fix extreme != selectivity for NDV=1
The original selectivity of 1.0 - 1.0/ndv makes sense for
large NDVs, but the result is 0.0 in case of ndv==1, which
leads to a cardinality of 1 even for huge tables. The new
selectivity is 0.5.
Note that as the formula for = is not changed (1.0/ndv),
NOT col="const" will still lead to 0.0 selectivity if ndv=1.
Changing the formula of NOT or = would have caused a lot of
subtle changes in plans in tests, so I don't want to touch
those before coming to wider agreement about the correct
approach.
IMPALA-7601 contains some discussion about these formulas.
Testing:
- added a regression test
Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
2 files changed, 12 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/18543/4
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
[Impala-ASF-CR] IMPALA-11301: Fix extreme != selectivity for NDV=1
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18543 )
Change subject: IMPALA-11301: Fix extreme != selectivity for NDV=1
......................................................................
Patch Set 4:
> Latest changes lgtm. Only other thought was about the unit test
> itself. It relies on the current behavior of row_number ndv
> estimate. Were you thinking of adding any other test against a
> table whose column had ndv=1 ?
Add a test with a table with a column of ndv 1. Originally I thought that there are no such test tables and didn't want to create one, but the jenkins job has pointed me to tpcds tables.
--
To view, visit http://gerrit.cloudera.org:8080/18543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6b5334a8d7d6ca46a450ff98ae03e5269faaa3c6
Gerrit-Change-Number: 18543
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Aman Sinha <am...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Gabor Kaszab <ga...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Comment-Date: Thu, 19 May 2022 17:34:04 +0000
Gerrit-HasComments: No