You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xzj7019 (via GitHub)" <gi...@apache.org> on 2024/09/11 03:28:39 UTC
[PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
xzj7019 opened a new pull request, #40654:
URL: https://github.com/apache/doris/pull/40654
## Proposed changes
Stats deriving refinement step 1: clean count usage in ColumnStatistic during stats deriving, to avoid serious stats deriving problem.
a. use Statistics rowCount instead of count in ColumnStatistic in stats deriving, since these two info may be inconsistent and lead to stats deriving problem.
b. remove setCount interface to avoid use this count field during deriving in the future.
c. refine a tpcds case plan by fixing a wrong row count info using in original ColumnStatistic count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753867317
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
if (cond instanceof EqualTo) {
EqualTo equal = (EqualTo) cond;
if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
- ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
- if (buildColStats == null) {
- buildColStats = buildStats.findColumnStatistics(equal.right());
+ Slot buildSideSlot = null;
+ if (buildStats.findColumnStatistics(equal.left()) != null) {
+ buildSideSlot = (Slot) equal.left();
+ } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+ buildSideSlot = (Slot) equal.right();
}
- if (buildColStats != null) {
- double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
- buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
- sel = Math.min(sel, buildSel);
+ // the processing is mainly for olap table since external table rarely has column level statistics
Review Comment:
yes, i will think a todo solution to resolve external table problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357998653
<details>
<summary>ClickBench: <b>Total hot run time: 32.42 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 71b338fb5d331bf25331f1674e07d2edad9665a4, data reload: false
query1 0.05 0.04 0.04
query2 0.07 0.03 0.02
query3 0.23 0.06 0.06
query4 1.66 0.10 0.10
query5 0.52 0.50 0.50
query6 1.12 0.72 0.72
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.56 0.49 0.50
query10 0.54 0.56 0.53
query11 0.14 0.10 0.11
query12 0.13 0.11 0.11
query13 0.60 0.59 0.60
query14 3.00 2.99 2.96
query15 0.88 0.83 0.83
query16 0.39 0.40 0.38
query17 1.04 1.03 1.02
query18 0.21 0.21 0.21
query19 1.95 1.83 1.99
query20 0.01 0.01 0.01
query21 15.35 0.61 0.60
query22 2.83 2.64 1.52
query23 17.57 0.84 0.71
query24 3.09 1.31 0.95
query25 0.28 0.12 0.10
query26 0.54 0.15 0.14
query27 0.03 0.04 0.04
query28 10.64 1.10 1.07
query29 12.55 3.27 3.26
query30 0.25 0.06 0.05
query31 2.88 0.38 0.39
query32 3.26 0.46 0.46
query33 2.99 3.00 3.02
query34 16.79 4.39 4.39
query35 4.48 4.42 4.42
query36 0.69 0.49 0.50
query37 0.09 0.05 0.06
query38 0.04 0.03 0.04
query39 0.03 0.02 0.02
query40 0.16 0.12 0.12
query41 0.08 0.02 0.02
query42 0.03 0.03 0.02
query43 0.03 0.04 0.04
Total cold run time: 107.84 s
Total hot run time: 32.42 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754156266
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -1411,7 +1404,7 @@ private Statistics computeWindow(Window windowOperator) {
private ColumnStatistic unionColumn(ColumnStatistic leftStats, double leftRowCount, ColumnStatistic rightStats,
double rightRowCount, DataType dataType) {
if (leftStats.isUnKnown() || rightStats.isUnKnown()) {
- return new ColumnStatisticBuilder(leftStats).setCount(leftRowCount + rightRowCount).build();
+ return new ColumnStatisticBuilder(leftStats).build();
Review Comment:
row count info will be updated by Statistics and no need to maintain it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753867317
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
if (cond instanceof EqualTo) {
EqualTo equal = (EqualTo) cond;
if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
- ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
- if (buildColStats == null) {
- buildColStats = buildStats.findColumnStatistics(equal.right());
+ Slot buildSideSlot = null;
+ if (buildStats.findColumnStatistics(equal.left()) != null) {
+ buildSideSlot = (Slot) equal.left();
+ } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+ buildSideSlot = (Slot) equal.right();
}
- if (buildColStats != null) {
- double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
- buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
- sel = Math.min(sel, buildSel);
+ // the processing is mainly for olap table since external table rarely has column level statistics
Review Comment:
yes, i will think out a todo solution to refine external table problem(it seems ok now since this function is only invoked by semi join estimation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360054838
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1766629841
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java:
##########
@@ -56,9 +56,24 @@ public ColumnStatisticBuilder(ColumnStatistic columnStatistic) {
this.updatedTime = columnStatistic.updatedTime;
}
- public ColumnStatisticBuilder setCount(double count) {
+ // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+ public ColumnStatisticBuilder(double count) {
Review Comment:
it is used to compatible with current code in the initialization stage, but not in the deriving stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #40654:
URL: https://github.com/apache/doris/pull/40654
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357378816
<details>
<summary>TPC-H: <b>Total hot run time: 41201 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c695d54577d75b68b1a3c97de44f540fa6e89de, data reload: false
------ Round 1 ----------------------------------
q1 17583 7341 7277 7277
q2 2041 160 161 160
q3 10670 1066 1183 1066
q4 10559 728 763 728
q5 7753 3041 3085 3041
q6 234 148 145 145
q7 994 617 596 596
q8 9433 2013 2053 2013
q9 6818 6430 6395 6395
q10 7051 2279 2304 2279
q11 452 248 252 248
q12 407 211 219 211
q13 17804 2984 2972 2972
q14 239 218 221 218
q15 577 525 528 525
q16 691 614 619 614
q17 983 772 766 766
q18 7303 6735 6581 6581
q19 1443 1044 1000 1000
q20 589 293 269 269
q21 4019 3073 3133 3073
q22 1100 1027 1024 1024
Total cold run time: 108743 ms
Total hot run time: 41201 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7258 7243 7186 7186
q2 336 222 234 222
q3 2934 2917 2990 2917
q4 2023 1864 1844 1844
q5 5578 5624 5554 5554
q6 230 145 143 143
q7 2235 1824 1780 1780
q8 3254 3423 3402 3402
q9 8720 8870 8719 8719
q10 3547 3389 3443 3389
q11 595 488 486 486
q12 794 597 604 597
q13 10292 3178 3183 3178
q14 307 300 260 260
q15 589 534 530 530
q16 721 693 662 662
q17 1777 1581 1584 1581
q18 8156 7819 7808 7808
q19 1748 1593 1461 1461
q20 2136 1882 1879 1879
q21 5602 5237 5421 5237
q22 1194 1036 1037 1036
Total cold run time: 70026 ms
Total hot run time: 59871 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753891100
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
if (cond instanceof EqualTo) {
EqualTo equal = (EqualTo) cond;
if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
- ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
- if (buildColStats == null) {
- buildColStats = buildStats.findColumnStatistics(equal.right());
+ Slot buildSideSlot = null;
+ if (buildStats.findColumnStatistics(equal.left()) != null) {
+ buildSideSlot = (Slot) equal.left();
+ } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+ buildSideSlot = (Slot) equal.right();
}
- if (buildColStats != null) {
- double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
- buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
- sel = Math.min(sel, buildSel);
+ // the processing is mainly for olap table since external table rarely has column level statistics
+ if (buildSideSlot instanceof SlotReference
+ && ((SlotReference) buildSideSlot).getTable().isPresent()
+ && (((SlotReference) buildSideSlot).getTable().get() instanceof OlapTable)) {
+ OlapTable olapTable = (OlapTable) ((SlotReference) buildSideSlot).getTable().get();
+ TableStatsMeta tableStatsMeta = Env.getCurrentEnv().getAnalysisManager()
+ .findTableStatsStatus(olapTable.getId());
+ if (tableStatsMeta != null) {
+ double tableRowCount = tableStatsMeta.getRowCount(olapTable.getBaseIndexId());
Review Comment:
1, 3: yes but seems no other way to do if we will remove the count usage. I will do a next pr to split no-stats and stats-available path to allow count 's usage in no-stats cases.
2: I will move total handling logic from row count acquirement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753812173
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java:
##########
@@ -56,9 +56,24 @@ public ColumnStatisticBuilder(ColumnStatistic columnStatistic) {
this.updatedTime = columnStatistic.updatedTime;
}
- public ColumnStatisticBuilder setCount(double count) {
+ // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+ public ColumnStatisticBuilder(double count) {
this.count = count;
- return this;
+ }
+
+ public ColumnStatisticBuilder(ColumnStatistic columnStatistic, double count) {
Review Comment:
it can be used by cache initialization and specially set rowcount, not only UNKNOWN.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2359862594
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357956982
<details>
<summary>TPC-H: <b>Total hot run time: 41210 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 71b338fb5d331bf25331f1674e07d2edad9665a4, data reload: false
------ Round 1 ----------------------------------
q1 18178 7518 7315 7315
q2 2101 168 186 168
q3 10927 1138 1189 1138
q4 10208 716 702 702
q5 7726 3042 3049 3042
q6 233 151 146 146
q7 999 609 601 601
q8 9424 2072 2018 2018
q9 6790 6351 6378 6351
q10 7011 2295 2291 2291
q11 441 251 249 249
q12 413 214 216 214
q13 17777 2954 3032 2954
q14 237 211 226 211
q15 575 522 532 522
q16 682 612 617 612
q17 980 819 762 762
q18 7219 6559 6622 6559
q19 1395 953 1031 953
q20 598 301 295 295
q21 3962 3182 3103 3103
q22 1121 1016 1004 1004
Total cold run time: 108997 ms
Total hot run time: 41210 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7275 7197 7173 7173
q2 328 235 237 235
q3 2940 2797 2802 2797
q4 1925 1636 1687 1636
q5 5363 5454 5455 5454
q6 226 143 141 141
q7 2097 1719 1695 1695
q8 3147 3313 3321 3313
q9 8411 8395 8444 8395
q10 3373 3340 3310 3310
q11 570 470 471 470
q12 771 607 596 596
q13 9559 3011 3008 3008
q14 292 264 261 261
q15 558 507 499 499
q16 715 667 680 667
q17 1760 1566 1530 1530
q18 7736 7255 7374 7255
q19 1665 1566 1493 1493
q20 2059 1805 1835 1805
q21 5408 5189 5186 5186
q22 1129 1012 981 981
Total cold run time: 67307 ms
Total hot run time: 57900 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357913739
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754158093
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatisticBuilder.java:
##########
@@ -50,9 +50,23 @@ public PartitionColumnStatisticBuilder(PartitionColumnStatistic statistic) {
this.updatedTime = statistic.updatedTime;
}
- public PartitionColumnStatisticBuilder setCount(double count) {
+ // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+ public PartitionColumnStatisticBuilder(double count) {
this.count = count;
- return this;
+ }
+
+ public PartitionColumnStatisticBuilder(PartitionColumnStatistic partitionColumnStatistic, double count) {
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753609665
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
if (cond instanceof EqualTo) {
EqualTo equal = (EqualTo) cond;
if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
- ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
- if (buildColStats == null) {
- buildColStats = buildStats.findColumnStatistics(equal.right());
+ Slot buildSideSlot = null;
+ if (buildStats.findColumnStatistics(equal.left()) != null) {
+ buildSideSlot = (Slot) equal.left();
+ } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+ buildSideSlot = (Slot) equal.right();
}
- if (buildColStats != null) {
- double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
- buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
- sel = Math.min(sel, buildSel);
+ // the processing is mainly for olap table since external table rarely has column level statistics
Review Comment:
1. if this funciton does not work for external table, put this check at the begining of this function.
2. It seems not good if the sel for external table is always 1
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java:
##########
@@ -56,9 +56,24 @@ public ColumnStatisticBuilder(ColumnStatistic columnStatistic) {
this.updatedTime = columnStatistic.updatedTime;
}
- public ColumnStatisticBuilder setCount(double count) {
+ // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+ public ColumnStatisticBuilder(double count) {
this.count = count;
- return this;
+ }
+
+ public ColumnStatisticBuilder(ColumnStatistic columnStatistic, double count) {
Review Comment:
this constructor can be removed. the parameter columnStatistic is always UNKNOWN.
It can be replaced by ColumnStatisticBuilder(count)
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -403,8 +403,7 @@ private Statistics computeOlapScan(OlapScan olapScan) {
for (Slot slot : ((Relation) olapScan).getOutput()) {
if (derivedStats.findColumnStatistics(slot) == null) {
derivedStats.addColumnStats(slot,
- new ColumnStatisticBuilder(ColumnStatistic.UNKNOWN)
- .setCount(derivedRowCount).build());
+ new ColumnStatisticBuilder(ColumnStatistic.UNKNOWN, derivedRowCount).build());
Review Comment:
new ColumnStatisticBuilder(derivedRowCount).build()
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
if (cond instanceof EqualTo) {
EqualTo equal = (EqualTo) cond;
if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
- ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
- if (buildColStats == null) {
- buildColStats = buildStats.findColumnStatistics(equal.right());
+ Slot buildSideSlot = null;
+ if (buildStats.findColumnStatistics(equal.left()) != null) {
+ buildSideSlot = (Slot) equal.left();
+ } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+ buildSideSlot = (Slot) equal.right();
}
- if (buildColStats != null) {
- double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
- buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
- sel = Math.min(sel, buildSel);
+ // the processing is mainly for olap table since external table rarely has column level statistics
+ if (buildSideSlot instanceof SlotReference
+ && ((SlotReference) buildSideSlot).getTable().isPresent()
+ && (((SlotReference) buildSideSlot).getTable().get() instanceof OlapTable)) {
+ OlapTable olapTable = (OlapTable) ((SlotReference) buildSideSlot).getTable().get();
+ TableStatsMeta tableStatsMeta = Env.getCurrentEnv().getAnalysisManager()
+ .findTableStatsStatus(olapTable.getId());
+ if (tableStatsMeta != null) {
+ double tableRowCount = tableStatsMeta.getRowCount(olapTable.getBaseIndexId());
Review Comment:
1. the base table row count is usually much more than that of rollup.
2. tableStatsMeta.getRowCount(olapTable.getBaseIndexId()) may return -1, if the table is not analyzed. In fact, BE reported rowCount has higher priority than that in TableMeta.
3. the assumption: "this slot is a base table slot” is easiy broken by alias. for example " select .. from (select A as x from T) T1
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -1411,7 +1404,7 @@ private Statistics computeWindow(Window windowOperator) {
private ColumnStatistic unionColumn(ColumnStatistic leftStats, double leftRowCount, ColumnStatistic rightStats,
double rightRowCount, DataType dataType) {
if (leftStats.isUnKnown() || rightStats.isUnKnown()) {
- return new ColumnStatisticBuilder(leftStats).setCount(leftRowCount + rightRowCount).build();
+ return new ColumnStatisticBuilder(leftStats).build();
Review Comment:
return new ColumnStatisticBuilder(leftStats.count + rightStats.count).build();
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java:
##########
@@ -323,6 +323,8 @@ private Statistics estimateEqualTo(ComparisonPredicate cp, ColumnStatistic stats
selectivity = DEFAULT_INEQUALITY_COEFFICIENT;
} else {
double ndv = statsForLeft.ndv;
+ double numNulls = statsForLeft.numNulls;
Review Comment:
numNulls = statsForLeft.numNulls * statsForRight.ndv
Usually, this func is used to evaluate join condition: T1.A=T2.B
the context.statistics is in charge of T1 cross join T2, and hence, the number of tuples, in which A=null or B=null, is A.numNulls * B.numNulls
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatistic.java:
##########
@@ -38,14 +38,14 @@ public class PartitionColumnStatistic {
private static final Logger LOG = LogManager.getLogger(PartitionColumnStatistic.class);
- public static PartitionColumnStatistic UNKNOWN = new PartitionColumnStatisticBuilder().setAvgSizeByte(1)
- .setNdv(new Hll128()).setNumNulls(1).setCount(1).setMaxValue(Double.POSITIVE_INFINITY)
+ public static PartitionColumnStatistic UNKNOWN = new PartitionColumnStatisticBuilder(1).setAvgSizeByte(1)
+ .setNdv(new Hll128()).setNumNulls(1).setMaxValue(Double.POSITIVE_INFINITY)
.setMinValue(Double.NEGATIVE_INFINITY)
.setIsUnknown(true).setUpdatedTime("")
.build();
- public static PartitionColumnStatistic ZERO = new PartitionColumnStatisticBuilder().setAvgSizeByte(0)
- .setNdv(new Hll128()).setNumNulls(0).setCount(0).setMaxValue(Double.NaN).setMinValue(Double.NaN)
+ public static PartitionColumnStatistic ZERO = new PartitionColumnStatisticBuilder(0).setAvgSizeByte(0)
Review Comment:
not used any more. Please remove it
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatistic.java:
##########
@@ -197,7 +198,6 @@ public ColumnStatistic updateByLimit(long limit, double rowCount) {
}
double newNdv = Math.ceil(Math.min(ndv, limit));
return new ColumnStatisticBuilder()
- .setCount(Math.ceil(limit))
Review Comment:
the function "updateByLimit" can be removed.
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatisticBuilder.java:
##########
@@ -50,9 +50,23 @@ public PartitionColumnStatisticBuilder(PartitionColumnStatistic statistic) {
this.updatedTime = statistic.updatedTime;
}
- public PartitionColumnStatisticBuilder setCount(double count) {
+ // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+ public PartitionColumnStatisticBuilder(double count) {
this.count = count;
- return this;
+ }
+
+ public PartitionColumnStatisticBuilder(PartitionColumnStatistic partitionColumnStatistic, double count) {
Review Comment:
partitionColumnStatistic is always unkown, remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1759707263
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -403,8 +403,7 @@ private Statistics computeOlapScan(OlapScan olapScan) {
for (Slot slot : ((Relation) olapScan).getOutput()) {
if (derivedStats.findColumnStatistics(slot) == null) {
derivedStats.addColumnStats(slot,
- new ColumnStatisticBuilder(ColumnStatistic.UNKNOWN)
- .setCount(derivedRowCount).build());
+ new ColumnStatisticBuilder(ColumnStatistic.UNKNOWN, derivedRowCount).build());
Review Comment:
remain it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342545367
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342867779
<details>
<summary>TPC-H: <b>Total hot run time: 38407 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f46762ab2927b314067608dd3befc9f98359a07, data reload: false
------ Round 1 ----------------------------------
q1 18315 4478 4400 4400
q2 2586 190 199 190
q3 11027 1174 1183 1174
q4 11035 875 756 756
q5 8089 2931 2839 2839
q6 233 141 138 138
q7 976 632 608 608
q8 9328 2074 2075 2074
q9 7329 6562 6548 6548
q10 7016 2265 2262 2262
q11 439 238 240 238
q12 394 222 224 222
q13 17756 3122 3112 3112
q14 284 233 240 233
q15 544 487 483 483
q16 520 437 436 436
q17 975 647 744 647
q18 7358 7006 6822 6822
q19 1386 1003 1051 1003
q20 718 346 322 322
q21 3935 3073 2870 2870
q22 1152 1030 1047 1030
Total cold run time: 111395 ms
Total hot run time: 38407 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4370 4297 4270 4270
q2 377 266 270 266
q3 2948 2672 2698 2672
q4 1936 1686 1742 1686
q5 5416 5419 5452 5419
q6 225 132 130 130
q7 2149 1795 1750 1750
q8 3205 3377 3337 3337
q9 8448 8478 8470 8470
q10 3494 3205 3234 3205
q11 610 528 501 501
q12 798 635 587 587
q13 8525 3072 3098 3072
q14 300 295 279 279
q15 509 484 492 484
q16 528 499 504 499
q17 1794 1494 1473 1473
q18 7833 7559 7457 7457
q19 1673 1463 1499 1463
q20 2092 1843 1848 1843
q21 5511 5276 5433 5276
q22 1169 1095 1019 1019
Total cold run time: 63910 ms
Total hot run time: 55158 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754155446
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatistic.java:
##########
@@ -197,7 +198,6 @@ public ColumnStatistic updateByLimit(long limit, double rowCount) {
}
double newNdv = Math.ceil(Math.min(ndv, limit));
return new ColumnStatisticBuilder()
- .setCount(Math.ceil(limit))
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2343481893
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357460686
<details>
<summary>TPC-DS: <b>Total hot run time: 195136 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6d12244fa198a7188aa59b981c7818afb9c2f078, data reload: false
query1 955 366 380 366
query2 6551 1986 2038 1986
query3 6704 210 230 210
query4 34467 23477 23490 23477
query5 4363 481 469 469
query6 270 173 175 173
query7 4633 298 308 298
query8 290 226 229 226
query9 9657 2669 2656 2656
query10 450 283 274 274
query11 18170 15210 15248 15210
query12 152 101 97 97
query13 1628 433 394 394
query14 10390 7398 7356 7356
query15 259 168 175 168
query16 8118 415 461 415
query17 1657 551 553 551
query18 2127 291 296 291
query19 356 145 144 144
query20 116 108 106 106
query21 215 102 101 101
query22 4369 4236 4068 4068
query23 34763 34415 34310 34310
query24 11201 2863 2871 2863
query25 638 393 377 377
query26 1201 155 158 155
query27 2485 280 281 280
query28 7814 2450 2443 2443
query29 836 413 411 411
query30 328 162 157 157
query31 1026 780 828 780
query32 101 55 58 55
query33 758 291 312 291
query34 961 494 495 494
query35 848 768 724 724
query36 1077 932 931 931
query37 165 89 87 87
query38 3995 3898 3868 3868
query39 1466 1473 1411 1411
query40 207 93 94 93
query41 51 45 46 45
query42 121 98 95 95
query43 522 468 479 468
query44 1251 821 805 805
query45 194 162 165 162
query46 1150 766 775 766
query47 1891 1798 1814 1798
query48 462 353 381 353
query49 1157 413 391 391
query50 813 398 404 398
query51 7023 6820 6965 6820
query52 98 85 85 85
query53 255 181 186 181
query54 1160 460 455 455
query55 81 80 75 75
query56 267 261 256 256
query57 1219 1087 1073 1073
query58 234 229 225 225
query59 3177 2954 2779 2779
query60 297 253 277 253
query61 105 102 103 102
query62 841 681 652 652
query63 213 191 185 185
query64 4241 641 695 641
query65 3262 3225 3187 3187
query66 1445 313 316 313
query67 16164 15503 15463 15463
query68 3113 872 863 863
query69 458 361 353 353
query70 1193 1169 1184 1169
query71 350 358 344 344
query72 6146 3579 3557 3557
query73 599 587 589 587
query74 9288 8975 9008 8975
query75 3097 2889 2937 2889
query76 1942 905 891 891
query77 428 378 383 378
query78 9411 9225 9345 9225
query79 936 893 874 874
query80 621 621 600 600
query81 458 262 258 258
query82 244 233 251 233
query83 172 169 170 169
query84 246 118 109 109
query85 745 436 508 436
query86 365 320 310 310
query87 4280 4343 4330 4330
query88 4470 4129 4079 4079
query89 380 358 358 358
query90 1454 325 320 320
query91 182 167 166 166
query92 82 76 73 73
query93 920 913 902 902
query94 567 349 356 349
query95 433 418 420 418
query96 501 486 488 486
query97 3076 3086 3112 3086
query98 239 233 230 230
query99 1402 1335 1327 1327
Total cold run time: 291259 ms
Total hot run time: 195136 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1766085281
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/ExpressionEstimation.java:
##########
@@ -274,13 +274,13 @@ public ColumnStatistic visitBinaryArithmetic(BinaryArithmetic binaryArithmetic,
int exprResultTypeWidth = binaryArithmetic.getDataType().width();
double dataSize = exprResultTypeWidth * rowCount;
if (binaryArithmetic instanceof Add) {
- return new ColumnStatisticBuilder().setCount(rowCount).setNdv(ndv).setAvgSizeByte(leftColStats.avgSizeByte)
Review Comment:
why not set count? count will be 0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360200985
<details>
<summary>ClickBench: <b>Total hot run time: 32.82 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ad5c8868b2e5c697c27fc0f217dccdb16c4bf5ec, data reload: false
query1 0.05 0.04 0.04
query2 0.07 0.03 0.03
query3 0.23 0.07 0.07
query4 1.64 0.09 0.10
query5 0.50 0.49 0.51
query6 1.16 0.74 0.72
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.56 0.50 0.48
query10 0.55 0.55 0.53
query11 0.16 0.10 0.10
query12 0.13 0.11 0.12
query13 0.62 0.59 0.59
query14 2.98 3.14 3.10
query15 0.91 0.82 0.81
query16 0.39 0.37 0.40
query17 1.06 1.04 1.08
query18 0.21 0.20 0.19
query19 1.96 1.87 2.02
query20 0.01 0.01 0.01
query21 15.35 0.59 0.57
query22 3.26 2.43 2.07
query23 17.42 0.82 0.78
query24 2.47 0.80 1.86
query25 0.27 0.05 0.07
query26 0.40 0.15 0.13
query27 0.04 0.05 0.03
query28 10.51 1.10 1.07
query29 12.55 3.22 3.22
query30 0.25 0.06 0.05
query31 2.87 0.39 0.38
query32 3.28 0.47 0.46
query33 3.03 3.02 3.01
query34 17.03 4.37 4.32
query35 4.40 4.42 4.44
query36 0.68 0.48 0.51
query37 0.08 0.06 0.07
query38 0.04 0.03 0.03
query39 0.03 0.02 0.02
query40 0.16 0.13 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 107.52 s
Total hot run time: 32.82 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360166826
<details>
<summary>TPC-H: <b>Total hot run time: 41925 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ad5c8868b2e5c697c27fc0f217dccdb16c4bf5ec, data reload: false
------ Round 1 ----------------------------------
q1 17748 7373 7290 7290
q2 2049 161 156 156
q3 10650 1192 1167 1167
q4 10228 750 818 750
q5 7769 3165 3101 3101
q6 234 150 148 148
q7 1011 630 615 615
q8 9449 2074 2111 2074
q9 6868 6430 6438 6430
q10 7043 2265 2270 2265
q11 437 244 247 244
q12 407 229 230 229
q13 17789 3009 3035 3009
q14 265 217 229 217
q15 586 531 530 530
q16 685 622 609 609
q17 992 873 836 836
q18 7354 6656 6796 6656
q19 1401 959 960 959
q20 581 279 292 279
q21 4063 3348 3588 3348
q22 1140 1013 1018 1013
Total cold run time: 108749 ms
Total hot run time: 41925 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7306 7246 7243 7243
q2 331 233 229 229
q3 3090 2985 2987 2985
q4 2165 1840 1807 1807
q5 5680 5625 5707 5625
q6 231 139 147 139
q7 2291 1885 1806 1806
q8 3351 3488 3472 3472
q9 8704 8851 8798 8798
q10 3519 3490 3566 3490
q11 584 473 469 469
q12 808 568 606 568
q13 9395 3221 3232 3221
q14 313 268 281 268
q15 567 522 533 522
q16 726 691 676 676
q17 1844 1629 1570 1570
q18 8418 7739 7721 7721
q19 1692 1542 1597 1542
q20 2114 1882 1886 1882
q21 5563 5465 5469 5465
q22 1134 1074 1051 1051
Total cold run time: 69826 ms
Total hot run time: 60549 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753867317
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
if (cond instanceof EqualTo) {
EqualTo equal = (EqualTo) cond;
if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
- ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
- if (buildColStats == null) {
- buildColStats = buildStats.findColumnStatistics(equal.right());
+ Slot buildSideSlot = null;
+ if (buildStats.findColumnStatistics(equal.left()) != null) {
+ buildSideSlot = (Slot) equal.left();
+ } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+ buildSideSlot = (Slot) equal.right();
}
- if (buildColStats != null) {
- double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
- buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
- sel = Math.min(sel, buildSel);
+ // the processing is mainly for olap table since external table rarely has column level statistics
Review Comment:
yes, i will think out a todo solution to resolve external table problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754152953
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatistic.java:
##########
@@ -38,14 +38,14 @@ public class PartitionColumnStatistic {
private static final Logger LOG = LogManager.getLogger(PartitionColumnStatistic.class);
- public static PartitionColumnStatistic UNKNOWN = new PartitionColumnStatisticBuilder().setAvgSizeByte(1)
- .setNdv(new Hll128()).setNumNulls(1).setCount(1).setMaxValue(Double.POSITIVE_INFINITY)
+ public static PartitionColumnStatistic UNKNOWN = new PartitionColumnStatisticBuilder(1).setAvgSizeByte(1)
+ .setNdv(new Hll128()).setNumNulls(1).setMaxValue(Double.POSITIVE_INFINITY)
.setMinValue(Double.NEGATIVE_INFINITY)
.setIsUnknown(true).setUpdatedTime("")
.build();
- public static PartitionColumnStatistic ZERO = new PartitionColumnStatisticBuilder().setAvgSizeByte(0)
- .setNdv(new Hll128()).setNumNulls(0).setCount(0).setMaxValue(Double.NaN).setMinValue(Double.NaN)
+ public static PartitionColumnStatistic ZERO = new PartitionColumnStatisticBuilder(0).setAvgSizeByte(0)
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357426385
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1766117795
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatistic.java:
##########
@@ -46,20 +46,18 @@ public class ColumnStatistic {
private static final Logger LOG = LogManager.getLogger(ColumnStatistic.class);
- public static ColumnStatistic UNKNOWN = new ColumnStatisticBuilder().setAvgSizeByte(1).setNdv(1)
- .setNumNulls(1).setCount(1).setMaxValue(Double.POSITIVE_INFINITY).setMinValue(Double.NEGATIVE_INFINITY)
+ public static ColumnStatistic UNKNOWN = new ColumnStatisticBuilder(1).setAvgSizeByte(1).setNdv(1)
+ .setNumNulls(1).setMaxValue(Double.POSITIVE_INFINITY).setMinValue(Double.NEGATIVE_INFINITY)
.setIsUnknown(true).setUpdatedTime("")
.build();
- public static ColumnStatistic ZERO = new ColumnStatisticBuilder().setAvgSizeByte(0).setNdv(0)
- .setNumNulls(0).setCount(0).setMaxValue(Double.NaN).setMinValue(Double.NaN)
- .build();
-
public static final Set<Type> UNSUPPORTED_TYPE = Sets.newHashSet(
Type.HLL, Type.BITMAP, Type.ARRAY, Type.STRUCT, Type.MAP, Type.QUANTILE_STATE, Type.JSONB,
Type.VARIANT, Type.TIME, Type.TIMEV2, Type.LAMBDA_FUNCTION
);
+ // ATTENTION: Stats deriving WILL NOT use 'count' field any longer.
+ // Use 'rowCount' field in Statistics if needed.
Review Comment:
nit: if we don't use this field anymore, should we just remove it? remove field do not break metadata
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/ExpressionEstimation.java:
##########
@@ -274,13 +274,13 @@ public ColumnStatistic visitBinaryArithmetic(BinaryArithmetic binaryArithmetic,
int exprResultTypeWidth = binaryArithmetic.getDataType().width();
double dataSize = exprResultTypeWidth * rowCount;
if (binaryArithmetic instanceof Add) {
- return new ColumnStatisticBuilder().setCount(rowCount).setNdv(ndv).setAvgSizeByte(leftColStats.avgSizeByte)
Review Comment:
why not set count? count will be 0?
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatisticBuilder.java:
##########
@@ -56,9 +56,24 @@ public ColumnStatisticBuilder(ColumnStatistic columnStatistic) {
this.updatedTime = columnStatistic.updatedTime;
}
- public ColumnStatisticBuilder setCount(double count) {
+ // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+ public ColumnStatisticBuilder(double count) {
Review Comment:
if count in ColumnStatistic is not used anymore, why still have a ctor with count? need count to compute datasize?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357465279
<details>
<summary>ClickBench: <b>Total hot run time: 32.54 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6d12244fa198a7188aa59b981c7818afb9c2f078, data reload: false
query1 0.04 0.04 0.04
query2 0.07 0.03 0.02
query3 0.22 0.06 0.07
query4 1.65 0.10 0.10
query5 0.53 0.51 0.50
query6 1.15 0.74 0.72
query7 0.02 0.01 0.01
query8 0.03 0.03 0.03
query9 0.56 0.50 0.50
query10 0.56 0.55 0.53
query11 0.15 0.11 0.11
query12 0.13 0.10 0.11
query13 0.62 0.59 0.58
query14 3.05 2.96 2.98
query15 0.88 0.82 0.83
query16 0.39 0.39 0.39
query17 1.03 1.05 1.08
query18 0.22 0.20 0.20
query19 1.99 1.84 1.95
query20 0.01 0.01 0.01
query21 15.37 0.56 0.58
query22 2.62 2.69 1.69
query23 17.37 0.84 0.84
query24 2.93 0.96 0.84
query25 0.31 0.12 0.10
query26 0.29 0.14 0.14
query27 0.03 0.04 0.03
query28 10.90 1.10 1.06
query29 12.54 3.25 3.27
query30 0.24 0.06 0.05
query31 2.87 0.39 0.37
query32 3.28 0.46 0.47
query33 3.01 3.00 2.99
query34 16.91 4.40 4.40
query35 4.44 4.44 4.41
query36 0.67 0.48 0.48
query37 0.09 0.06 0.06
query38 0.05 0.03 0.04
query39 0.04 0.02 0.03
query40 0.17 0.12 0.12
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 107.58 s
Total hot run time: 32.54 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357982750
<details>
<summary>TPC-DS: <b>Total hot run time: 195290 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 71b338fb5d331bf25331f1674e07d2edad9665a4, data reload: false
query1 948 374 386 374
query2 6548 2051 1985 1985
query3 6704 209 223 209
query4 37232 23385 23470 23385
query5 4307 489 467 467
query6 267 178 183 178
query7 4629 302 312 302
query8 285 223 224 223
query9 9913 2690 2678 2678
query10 477 292 290 290
query11 18296 15171 15221 15171
query12 158 105 96 96
query13 1631 420 394 394
query14 11106 7863 7637 7637
query15 281 174 184 174
query16 7567 417 463 417
query17 1693 570 553 553
query18 1985 297 291 291
query19 339 141 148 141
query20 115 106 107 106
query21 207 105 105 105
query22 4472 4144 4311 4144
query23 34540 34214 33847 33847
query24 11238 2948 2932 2932
query25 599 386 404 386
query26 1152 161 167 161
query27 2621 282 285 282
query28 8137 2435 2412 2412
query29 806 402 409 402
query30 318 161 153 153
query31 1048 745 809 745
query32 91 58 56 56
query33 717 302 305 302
query34 911 503 513 503
query35 879 722 727 722
query36 1048 952 932 932
query37 153 89 125 89
query38 4029 3943 3899 3899
query39 1445 1427 1402 1402
query40 204 100 94 94
query41 49 47 47 47
query42 118 97 98 97
query43 514 477 467 467
query44 1217 811 797 797
query45 198 163 164 163
query46 1136 762 815 762
query47 1870 1801 1831 1801
query48 456 359 373 359
query49 1159 404 404 404
query50 839 409 423 409
query51 7135 6907 6975 6907
query52 105 90 88 88
query53 266 188 181 181
query54 1206 464 471 464
query55 111 81 80 80
query56 261 269 266 266
query57 1197 1103 1093 1093
query58 228 241 243 241
query59 3133 2817 2929 2817
query60 296 271 271 271
query61 108 101 98 98
query62 851 635 661 635
query63 226 192 188 188
query64 4066 664 643 643
query65 3288 3199 3192 3192
query66 1357 319 313 313
query67 16023 15522 15571 15522
query68 3046 887 874 874
query69 465 356 355 355
query70 1195 1210 1167 1167
query71 349 340 345 340
query72 5984 3575 3723 3575
query73 592 584 592 584
query74 9318 9017 8924 8924
query75 3080 2890 2863 2863
query76 1285 921 904 904
query77 397 387 389 387
query78 10305 10051 9298 9298
query79 945 922 911 911
query80 649 619 616 616
query81 457 261 257 257
query82 243 239 228 228
query83 171 168 167 167
query84 242 118 108 108
query85 732 430 540 430
query86 311 347 317 317
query87 4273 4429 4290 4290
query88 4461 4045 4025 4025
query89 372 362 367 362
query90 1471 320 322 320
query91 162 188 168 168
query92 80 75 75 75
query93 894 888 890 888
query94 578 400 394 394
query95 427 408 413 408
query96 482 489 479 479
query97 3142 3125 3157 3125
query98 232 227 226 226
query99 1399 1281 1289 1281
Total cold run time: 294248 ms
Total hot run time: 195290 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342828971
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1754158093
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatisticBuilder.java:
##########
@@ -50,9 +50,23 @@ public PartitionColumnStatisticBuilder(PartitionColumnStatistic statistic) {
this.updatedTime = statistic.updatedTime;
}
- public PartitionColumnStatisticBuilder setCount(double count) {
+ // ATTENTION: DON'T USE FOLLOWING TWO DURING STATS DERIVING EXCEPT FOR INITIALIZATION
+ public PartitionColumnStatisticBuilder(double count) {
this.count = count;
- return this;
+ }
+
+ public PartitionColumnStatisticBuilder(PartitionColumnStatistic partitionColumnStatistic, double count) {
Review Comment:
remain it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360192477
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1766627537
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/ColumnStatistic.java:
##########
@@ -46,20 +46,18 @@ public class ColumnStatistic {
private static final Logger LOG = LogManager.getLogger(ColumnStatistic.class);
- public static ColumnStatistic UNKNOWN = new ColumnStatisticBuilder().setAvgSizeByte(1).setNdv(1)
- .setNumNulls(1).setCount(1).setMaxValue(Double.POSITIVE_INFINITY).setMinValue(Double.NEGATIVE_INFINITY)
+ public static ColumnStatistic UNKNOWN = new ColumnStatisticBuilder(1).setAvgSizeByte(1).setNdv(1)
+ .setNumNulls(1).setMaxValue(Double.POSITIVE_INFINITY).setMinValue(Double.NEGATIVE_INFINITY)
.setIsUnknown(true).setUpdatedTime("")
.build();
- public static ColumnStatistic ZERO = new ColumnStatisticBuilder().setAvgSizeByte(0).setNdv(0)
- .setNumNulls(0).setCount(0).setMaxValue(Double.NaN).setMinValue(Double.NaN)
- .build();
-
public static final Set<Type> UNSUPPORTED_TYPE = Sets.newHashSet(
Type.HLL, Type.BITMAP, Type.ARRAY, Type.STRUCT, Type.MAP, Type.QUANTILE_STATE, Type.JSONB,
Type.VARIANT, Type.TIME, Type.TIMEV2, Type.LAMBDA_FUNCTION
);
+ // ATTENTION: Stats deriving WILL NOT use 'count' field any longer.
+ // Use 'rowCount' field in Statistics if needed.
Review Comment:
no-stats path may use it so field remains, and has-stats path and no-stats path will have different function/file in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357342636
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360008061
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2359862616
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2360190032
<details>
<summary>TPC-DS: <b>Total hot run time: 195087 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ad5c8868b2e5c697c27fc0f217dccdb16c4bf5ec, data reload: false
query1 1251 924 898 898
query2 6268 2122 2076 2076
query3 10809 4029 3798 3798
query4 68398 31474 23441 23441
query5 5164 472 460 460
query6 414 173 174 173
query7 5747 295 289 289
query8 316 226 223 223
query9 9276 2676 2621 2621
query10 493 287 286 286
query11 17697 15467 15868 15467
query12 157 112 99 99
query13 1588 429 405 405
query14 10988 7601 7605 7601
query15 207 170 176 170
query16 6726 486 478 478
query17 1062 561 552 552
query18 1130 302 325 302
query19 201 145 143 143
query20 108 105 104 104
query21 220 106 100 100
query22 4436 4343 4037 4037
query23 34177 33912 33885 33885
query24 6053 2875 2922 2875
query25 513 407 417 407
query26 671 162 160 160
query27 1732 285 291 285
query28 4214 2479 2422 2422
query29 662 426 419 419
query30 235 152 151 151
query31 986 795 813 795
query32 78 56 57 56
query33 430 297 303 297
query34 886 500 499 499
query35 844 726 730 726
query36 1082 950 943 943
query37 149 91 94 91
query38 4078 3838 3867 3838
query39 1487 1421 1399 1399
query40 206 99 100 99
query41 51 50 50 50
query42 116 93 94 93
query43 524 485 483 483
query44 1142 824 798 798
query45 198 165 163 163
query46 1165 751 747 747
query47 1902 1791 1892 1791
query48 454 360 358 358
query49 727 412 411 411
query50 861 399 401 399
query51 6955 6895 6933 6895
query52 102 87 85 85
query53 253 178 183 178
query54 574 481 451 451
query55 74 77 73 73
query56 269 257 258 257
query57 1205 1100 1104 1100
query58 215 215 230 215
query59 3264 2965 3069 2965
query60 299 270 269 269
query61 111 100 103 100
query62 778 670 652 652
query63 218 186 184 184
query64 1448 655 595 595
query65 3215 3182 3219 3182
query66 646 323 335 323
query67 15602 15473 15381 15381
query68 4452 595 564 564
query69 436 292 285 285
query70 1215 1175 1110 1110
query71 340 277 276 276
query72 6352 4087 3896 3896
query73 766 324 329 324
query74 9485 9025 9092 9025
query75 3333 2682 2693 2682
query76 1818 910 1028 910
query77 458 285 298 285
query78 9887 9147 9073 9073
query79 1066 550 531 531
query80 756 445 449 445
query81 483 255 242 242
query82 518 142 143 142
query83 156 135 141 135
query84 258 81 83 81
query85 776 292 277 277
query86 305 306 308 306
query87 4402 4371 4307 4307
query88 3485 2369 2356 2356
query89 390 273 284 273
query90 1988 193 191 191
query91 176 146 144 144
query92 58 46 46 46
query93 1073 525 532 525
query94 703 297 297 297
query95 342 254 262 254
query96 610 281 277 277
query97 3200 3070 3078 3070
query98 207 197 215 197
query99 1528 1314 1293 1293
Total cold run time: 314529 ms
Total hot run time: 195087 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342889265
<details>
<summary>TPC-DS: <b>Total hot run time: 192192 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f46762ab2927b314067608dd3befc9f98359a07, data reload: false
query1 915 385 375 375
query2 6460 1898 1866 1866
query3 6648 228 221 221
query4 33929 23434 23249 23249
query5 4163 508 498 498
query6 254 160 178 160
query7 4584 289 291 289
query8 301 228 225 225
query9 8462 2487 2481 2481
query10 432 286 260 260
query11 17308 15243 15021 15021
query12 151 99 97 97
query13 1634 394 367 367
query14 10008 6833 7179 6833
query15 283 172 167 167
query16 8002 471 468 468
query17 1562 568 546 546
query18 1967 307 281 281
query19 302 143 146 143
query20 118 110 110 110
query21 204 105 105 105
query22 4506 4440 4116 4116
query23 34155 33518 33608 33518
query24 11070 2863 2890 2863
query25 615 407 384 384
query26 1121 156 155 155
query27 2347 277 275 275
query28 7009 2057 2037 2037
query29 798 403 397 397
query30 315 161 147 147
query31 955 793 793 793
query32 95 53 54 53
query33 753 288 284 284
query34 993 470 477 470
query35 844 727 683 683
query36 1091 947 951 947
query37 163 88 85 85
query38 4027 3933 3850 3850
query39 1445 1450 1412 1412
query40 190 120 113 113
query41 50 47 46 46
query42 117 95 95 95
query43 485 473 476 473
query44 1187 767 736 736
query45 197 168 168 168
query46 1102 706 706 706
query47 1901 1805 1810 1805
query48 349 297 283 283
query49 1086 438 438 438
query50 819 401 409 401
query51 7068 7051 6796 6796
query52 105 89 92 89
query53 263 186 185 185
query54 1001 484 442 442
query55 77 74 74 74
query56 270 255 265 255
query57 1214 1069 1060 1060
query58 239 221 233 221
query59 2904 2713 2699 2699
query60 305 271 257 257
query61 108 102 99 99
query62 861 648 657 648
query63 228 204 189 189
query64 4290 706 676 676
query65 3266 3171 3151 3151
query66 1392 340 334 334
query67 15904 15771 15345 15345
query68 3132 856 833 833
query69 449 320 326 320
query70 1154 1198 1225 1198
query71 341 346 341 341
query72 6247 3455 3671 3455
query73 580 581 586 581
query74 8922 9043 9003 9003
query75 3193 2961 2946 2946
query76 1853 845 837 837
query77 474 456 406 406
query78 9426 9412 9203 9203
query79 911 881 846 846
query80 835 802 805 802
query81 453 254 256 254
query82 271 265 268 265
query83 193 193 202 193
query84 225 111 106 106
query85 657 430 389 389
query86 333 279 314 279
query87 4289 4488 4291 4291
query88 4185 4073 4027 4027
query89 375 367 368 367
query90 1517 321 311 311
query91 125 125 185 125
query92 80 76 74 74
query93 917 934 931 931
query94 610 360 344 344
query95 410 415 409 409
query96 467 469 470 469
query97 3134 3083 3103 3083
query98 231 227 233 227
query99 1415 1309 1262 1262
Total cold run time: 284371 ms
Total hot run time: 192192 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1753857864
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java:
##########
@@ -323,6 +323,8 @@ private Statistics estimateEqualTo(ComparisonPredicate cp, ColumnStatistic stats
selectivity = DEFAULT_INEQUALITY_COEFFICIENT;
} else {
double ndv = statsForLeft.ndv;
+ double numNulls = statsForLeft.numNulls;
Review Comment:
here is the t1.c1 = 1 like case and we already have the overall selectivity already and it should be adjusted by num of null, which is mainly for t1.c1. so the input pair of <numNulls, rowcount> is meaningful for t1.c1, and the context.statistics.rowcount is the replacement of original count of t1.c1's column statistics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2357450291
<details>
<summary>TPC-H: <b>Total hot run time: 41314 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6d12244fa198a7188aa59b981c7818afb9c2f078, data reload: false
------ Round 1 ----------------------------------
q1 17902 7413 7350 7350
q2 2563 160 159 159
q3 10922 1097 1155 1097
q4 10206 698 773 698
q5 7793 3104 3053 3053
q6 237 147 146 146
q7 995 610 611 610
q8 9417 2029 2053 2029
q9 6823 6350 6362 6350
q10 7004 2297 2283 2283
q11 430 239 248 239
q12 403 209 208 208
q13 17771 2953 2973 2953
q14 245 214 213 213
q15 580 521 520 520
q16 679 600 629 600
q17 969 846 778 778
q18 7211 6692 6680 6680
q19 1397 1020 958 958
q20 559 284 270 270
q21 4083 3141 3292 3141
q22 1109 985 979 979
Total cold run time: 109298 ms
Total hot run time: 41314 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7237 7216 7255 7216
q2 330 229 229 229
q3 2897 2810 2791 2791
q4 1925 1695 1751 1695
q5 5340 5390 5387 5387
q6 223 139 142 139
q7 2097 1709 1738 1709
q8 3150 3300 3315 3300
q9 8436 8348 8407 8348
q10 3406 3321 3321 3321
q11 576 472 474 472
q12 767 562 575 562
q13 6496 3004 2977 2977
q14 301 259 280 259
q15 550 522 501 501
q16 693 660 663 660
q17 1769 1553 1543 1543
q18 7787 7307 7303 7303
q19 1648 1462 1517 1462
q20 2059 1773 1801 1773
q21 5315 5252 5294 5252
q22 1108 1033 991 991
Total cold run time: 64110 ms
Total hot run time: 57890 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2358167594
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40654:
URL: https://github.com/apache/doris/pull/40654#issuecomment-2342900286
<details>
<summary>ClickBench: <b>Total hot run time: 31.13 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1f46762ab2927b314067608dd3befc9f98359a07, data reload: false
query1 0.04 0.04 0.04
query2 0.07 0.04 0.04
query3 0.22 0.05 0.05
query4 1.66 0.09 0.08
query5 0.49 0.49 0.50
query6 1.12 0.72 0.73
query7 0.02 0.01 0.01
query8 0.05 0.04 0.04
query9 0.55 0.49 0.49
query10 0.54 0.55 0.54
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.59 0.60 0.59
query14 1.44 1.41 1.42
query15 0.83 0.85 0.83
query16 0.36 0.36 0.37
query17 1.00 1.01 1.06
query18 0.20 0.21 0.20
query19 1.79 1.70 1.72
query20 0.02 0.01 0.01
query21 15.40 0.67 0.66
query22 4.26 7.22 1.54
query23 18.25 1.36 1.22
query24 2.17 0.22 0.21
query25 0.14 0.08 0.07
query26 0.28 0.18 0.19
query27 0.07 0.07 0.08
query28 13.18 1.03 1.00
query29 12.63 3.43 3.37
query30 0.25 0.05 0.05
query31 2.89 0.39 0.39
query32 3.25 0.48 0.48
query33 2.95 3.05 3.04
query34 17.16 4.45 4.42
query35 4.60 4.54 4.49
query36 0.66 0.49 0.49
query37 0.19 0.16 0.15
query38 0.15 0.16 0.15
query39 0.05 0.04 0.04
query40 0.16 0.13 0.13
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.04 0.05 0.04
Total cold run time: 110.18 s
Total hot run time: 31.13 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) clean count usage in ColumnStatistic during stats deriving [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on code in PR #40654:
URL: https://github.com/apache/doris/pull/40654#discussion_r1759707814
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/JoinEstimation.java:
##########
@@ -177,14 +181,27 @@ private static double computeSelectivityForBuildSideWhenColStatsUnknown(Statisti
if (cond instanceof EqualTo) {
EqualTo equal = (EqualTo) cond;
if (equal.left() instanceof Slot && equal.right() instanceof Slot) {
- ColumnStatistic buildColStats = buildStats.findColumnStatistics(equal.left());
- if (buildColStats == null) {
- buildColStats = buildStats.findColumnStatistics(equal.right());
+ Slot buildSideSlot = null;
+ if (buildStats.findColumnStatistics(equal.left()) != null) {
+ buildSideSlot = (Slot) equal.left();
+ } else if (buildStats.findColumnStatistics(equal.right()) != null) {
+ buildSideSlot = (Slot) equal.right();
}
- if (buildColStats != null) {
- double buildSel = Math.min(buildStats.getRowCount() / buildColStats.count, 1.0);
- buildSel = Math.max(buildSel, UNKNOWN_COL_STATS_FILTER_SEL_LOWER_BOUND);
- sel = Math.min(sel, buildSel);
+ // the processing is mainly for olap table since external table rarely has column level statistics
Review Comment:
it seems not easy to remove this part of count usage and since it is for no-stats, it can be remained for future use and the modification will be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org