You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2024/06/04 11:50:33 UTC
[PR] merge part stats [doris]
englefly opened a new pull request, #35875:
URL: https://github.com/apache/doris/pull/35875
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2160545358
<details>
<summary>TPC-DS: <b>Total hot run time: 172785 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d101cfd0875a21af3df33ce7b1b1d0103088825e, data reload: false
query1 949 383 359 359
query2 6441 2355 2326 2326
query3 6648 200 205 200
query4 19402 17422 17388 17388
query5 4116 444 450 444
query6 244 159 154 154
query7 4583 301 286 286
query8 320 290 278 278
query9 8543 2369 2317 2317
query10 583 293 271 271
query11 10380 10108 9954 9954
query12 140 83 80 80
query13 1655 392 384 384
query14 9961 7672 7616 7616
query15 244 186 175 175
query16 7791 270 258 258
query17 1902 518 526 518
query18 1903 276 277 276
query19 201 144 156 144
query20 92 82 82 82
query21 200 124 121 121
query22 4238 4034 4021 4021
query23 33590 32895 33024 32895
query24 11884 2784 2806 2784
query25 663 352 372 352
query26 1753 147 146 146
query27 2868 318 311 311
query28 7396 2021 2006 2006
query29 1073 616 599 599
query30 280 153 148 148
query31 946 741 728 728
query32 93 54 55 54
query33 773 285 276 276
query34 966 467 466 466
query35 758 625 605 605
query36 1090 964 949 949
query37 195 70 69 69
query38 2850 2733 2729 2729
query39 888 795 787 787
query40 271 122 126 122
query41 55 50 50 50
query42 118 97 97 97
query43 589 531 539 531
query44 1159 739 750 739
query45 199 168 171 168
query46 1075 733 699 699
query47 1857 1763 1760 1760
query48 378 293 299 293
query49 1214 413 397 397
query50 795 385 386 385
query51 6839 6683 6690 6683
query52 97 95 88 88
query53 355 286 285 285
query54 1009 446 451 446
query55 77 74 72 72
query56 271 247 249 247
query57 1170 1047 1050 1047
query58 279 232 248 232
query59 3498 3106 3128 3106
query60 288 256 263 256
query61 91 84 84 84
query62 653 441 439 439
query63 328 289 284 284
query64 9849 2270 1824 1824
query65 3170 3091 3076 3076
query66 1393 337 342 337
query67 15579 15263 14846 14846
query68 4599 539 547 539
query69 456 295 305 295
query70 1168 1072 1159 1072
query71 365 271 267 267
query72 7079 5341 5533 5341
query73 746 334 330 330
query74 5931 5604 5473 5473
query75 3348 2714 2672 2672
query76 2676 936 919 919
query77 497 310 318 310
query78 10342 9822 9763 9763
query79 2767 520 527 520
query80 1924 514 458 458
query81 595 218 221 218
query82 893 110 106 106
query83 312 165 170 165
query84 263 93 86 86
query85 1375 277 262 262
query86 472 337 336 336
query87 3279 3045 3074 3045
query88 3912 2434 2445 2434
query89 478 396 390 390
query90 1764 196 192 192
query91 129 98 95 95
query92 70 53 48 48
query93 1893 505 494 494
query94 1212 187 184 184
query95 405 327 324 324
query96 594 276 268 268
query97 3195 2993 3075 2993
query98 212 249 197 197
query99 1091 863 840 840
Total cold run time: 275300 ms
Total hot run time: 172785 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2162497892
<details>
<summary>TPC-H: <b>Total hot run time: 40456 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0cf12548e96986bab337a63ef39cf43c448a7e30, data reload: false
------ Round 1 ----------------------------------
q1 17627 4370 4305 4305
q2 2013 190 185 185
q3 10491 1175 1128 1128
q4 10210 814 720 720
q5 8536 2701 2674 2674
q6 222 140 139 139
q7 942 604 609 604
q8 9351 2102 2062 2062
q9 9043 6571 6536 6536
q10 8892 3706 3749 3706
q11 456 228 238 228
q12 535 222 220 220
q13 17912 3235 3197 3197
q14 270 228 235 228
q15 524 498 464 464
q16 543 410 397 397
q17 977 721 739 721
q18 8949 8069 7671 7671
q19 7912 1522 1448 1448
q20 673 322 319 319
q21 5037 3177 3885 3177
q22 382 330 327 327
Total cold run time: 121497 ms
Total hot run time: 40456 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4472 4251 4302 4251
q2 372 267 288 267
q3 3143 2910 2890 2890
q4 2026 1713 1645 1645
q5 5470 5457 5566 5457
q6 227 131 132 131
q7 2282 1818 1854 1818
q8 3232 3380 3388 3380
q9 8669 8816 8745 8745
q10 3973 3791 3839 3791
q11 606 509 495 495
q12 800 616 618 616
q13 17226 3108 3187 3108
q14 317 276 255 255
q15 546 492 483 483
q16 477 426 449 426
q17 1815 1517 1508 1508
q18 8169 7838 7764 7764
q19 1816 1652 1669 1652
q20 3632 1888 1845 1845
q21 10607 4777 4778 4777
q22 653 546 562 546
Total cold run time: 80530 ms
Total hot run time: 55850 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2162419066
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #35875:
URL: https://github.com/apache/doris/pull/35875#discussion_r1642690421
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -745,8 +747,32 @@ private ColumnStatistic getColumnStatistic(TableIf table, String colName, long i
return ColumnStatistic.UNKNOWN;
}
} else {
+ if (!partitionNames.isEmpty()) {
+ PartitionColumnStatisticBuilder builder = new PartitionColumnStatisticBuilder();
+ boolean isFirst = true;
+ boolean hasUnknown = false;
+ for (String partitionName : partitionNames) {
+ PartitionColumnStatistic pcolStats = Env.getCurrentEnv().getStatisticsCache()
+ .getPartitionColumnStatistics(
+ catalogId, dbId, table.getId(), idxId, partitionName, colName);
+ if (pcolStats.isUnKnown) {
+ hasUnknown = true;
+ break;
+ }
+ if (isFirst) {
+ builder = new PartitionColumnStatisticBuilder(pcolStats);
+ isFirst = false;
+ } else {
+ builder.merge(pcolStats);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] merge part stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2160025623
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2173303473
<details>
<summary>TPC-DS: <b>Total hot run time: 172376 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 41d2b015a273f3a2ede855950279317bb883abc3, data reload: false
query1 933 390 382 382
query2 6417 2359 2342 2342
query3 6616 198 196 196
query4 18738 17204 17321 17204
query5 3555 453 451 451
query6 237 163 157 157
query7 4576 305 293 293
query8 302 269 270 269
query9 8581 2352 2352 2352
query10 558 296 288 288
query11 10460 9961 10022 9961
query12 120 88 78 78
query13 1637 368 383 368
query14 9568 6505 7858 6505
query15 271 179 176 176
query16 7960 266 257 257
query17 1896 541 510 510
query18 1990 263 259 259
query19 188 149 153 149
query20 92 84 82 82
query21 207 119 123 119
query22 4256 4094 4067 4067
query23 33749 33652 33522 33522
query24 10923 2825 2993 2825
query25 571 356 389 356
query26 717 160 155 155
query27 2285 334 324 324
query28 6047 2096 2085 2085
query29 876 639 605 605
query30 244 153 152 152
query31 960 753 778 753
query32 95 54 55 54
query33 741 284 279 279
query34 1031 473 466 466
query35 743 639 631 631
query36 1143 981 979 979
query37 143 71 75 71
query38 2977 2926 2802 2802
query39 896 824 849 824
query40 222 133 124 124
query41 55 51 54 51
query42 136 98 108 98
query43 576 561 509 509
query44 1158 725 719 719
query45 189 160 162 160
query46 1065 706 721 706
query47 1827 1769 1780 1769
query48 398 299 295 295
query49 832 394 399 394
query50 774 379 377 377
query51 6848 6532 6627 6532
query52 103 93 92 92
query53 355 290 286 286
query54 909 425 427 425
query55 74 75 72 72
query56 274 244 263 244
query57 1126 1033 1046 1033
query58 250 261 233 233
query59 3209 3301 3292 3292
query60 284 268 270 268
query61 92 94 90 90
query62 593 436 426 426
query63 329 287 289 287
query64 8555 2216 1707 1707
query65 3132 3084 3114 3084
query66 741 323 324 323
query67 15357 15104 14919 14919
query68 4615 534 538 534
query69 591 414 353 353
query70 1170 1034 1139 1034
query71 427 266 275 266
query72 8052 5556 5242 5242
query73 730 319 321 319
query74 5884 5537 5536 5536
query75 3454 2624 2649 2624
query76 2476 910 879 879
query77 676 291 287 287
query78 10269 9921 9952 9921
query79 2204 512 509 509
query80 1032 457 452 452
query81 549 217 276 217
query82 645 99 100 99
query83 332 164 165 164
query84 255 81 82 81
query85 1806 278 265 265
query86 471 309 290 290
query87 3272 3088 3082 3082
query88 4199 2335 2336 2335
query89 460 375 373 373
query90 1818 195 186 186
query91 129 99 97 97
query92 61 48 49 48
query93 1929 497 492 492
query94 1198 186 181 181
query95 399 309 307 307
query96 587 268 264 264
query97 3216 3031 3049 3031
query98 210 201 190 190
query99 1105 855 863 855
Total cold run time: 266983 ms
Total hot run time: 172376 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #35875:
URL: https://github.com/apache/doris/pull/35875#discussion_r1636030225
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -745,8 +747,32 @@ private ColumnStatistic getColumnStatistic(TableIf table, String colName, long i
return ColumnStatistic.UNKNOWN;
}
} else {
+ if (!partitionNames.isEmpty()) {
+ PartitionColumnStatisticBuilder builder = new PartitionColumnStatisticBuilder();
+ boolean isFirst = true;
+ boolean hasUnknown = false;
+ for (String partitionName : partitionNames) {
+ PartitionColumnStatistic pcolStats = Env.getCurrentEnv().getStatisticsCache()
+ .getPartitionColumnStatistics(
+ catalogId, dbId, table.getId(), idxId, partitionName, colName);
+ if (pcolStats.isUnKnown) {
+ hasUnknown = true;
+ break;
+ }
+ if (isFirst) {
+ builder = new PartitionColumnStatisticBuilder(pcolStats);
+ isFirst = false;
+ } else {
+ builder.merge(pcolStats);
Review Comment:
if any pcol stats is unknown, the merge process is broken. this check is done in line758
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -764,19 +790,22 @@ private Statistics computeCatalogRelation(CatalogRelation catalogRelation) {
Set<SlotReference> slotSet = slotSetBuilder.build();
Map<Expression, ColumnStatisticBuilder> columnStatisticBuilderMap = new HashMap<>();
TableIf table = catalogRelation.getTable();
- boolean isOlapTable = table instanceof OlapTable;
AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
TableStatsMeta tableMeta = analysisManager.findTableStatsStatus(table.getId());
long tableUpdatedRows = tableMeta == null ? 0 : tableMeta.updatedRows.get();
- double rowCount = catalogRelation.getTable().getRowCountForNereids();
boolean hasUnknownKeyCol = false;
long idxId = -1;
+ List<String> selectedPartitionNames = new ArrayList<>();
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #35875:
URL: https://github.com/apache/doris/pull/35875
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2162533106
<details>
<summary>ClickBench: <b>Total hot run time: 30.45 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0cf12548e96986bab337a63ef39cf43c448a7e30, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.67 0.06 0.07
query5 0.50 0.49 0.48
query6 1.13 0.73 0.72
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.53 0.49 0.49
query10 0.53 0.53 0.54
query11 0.15 0.10 0.10
query12 0.14 0.12 0.12
query13 0.59 0.60 0.58
query14 0.76 0.77 0.77
query15 0.84 0.81 0.81
query16 0.37 0.36 0.36
query17 0.94 0.96 0.96
query18 0.23 0.23 0.24
query19 1.77 1.77 1.66
query20 0.01 0.01 0.01
query21 15.40 0.65 0.65
query22 4.15 7.21 1.78
query23 18.25 1.44 1.26
query24 2.12 0.22 0.21
query25 0.15 0.10 0.08
query26 0.27 0.17 0.18
query27 0.07 0.09 0.07
query28 13.23 1.01 1.01
query29 12.60 3.25 3.25
query30 0.25 0.06 0.05
query31 2.88 0.37 0.38
query32 3.28 0.47 0.46
query33 2.88 2.87 2.86
query34 17.05 4.71 4.63
query35 4.79 4.66 4.51
query36 0.64 0.47 0.49
query37 0.18 0.15 0.15
query38 0.15 0.14 0.15
query39 0.04 0.04 0.03
query40 0.17 0.14 0.14
query41 0.09 0.05 0.04
query42 0.05 0.05 0.04
query43 0.04 0.03 0.04
Total cold run time: 109.3 s
Total hot run time: 30.45 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2174788366
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2182075777
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2160455923
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2177398032
<details>
<summary>TPC-H: <b>Total hot run time: 39769 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 69f32fa800535c3026114840f838214f603a9b58, data reload: false
------ Round 1 ----------------------------------
q1 17606 4355 4416 4355
q2 2009 199 189 189
q3 10470 1122 1064 1064
q4 10206 761 737 737
q5 7457 2606 2621 2606
q6 217 134 129 129
q7 934 596 595 595
q8 9221 2048 2075 2048
q9 8929 6493 6513 6493
q10 8913 3731 3741 3731
q11 446 239 229 229
q12 486 227 223 223
q13 17777 2979 2975 2975
q14 276 231 214 214
q15 528 470 476 470
q16 535 373 370 370
q17 967 713 708 708
q18 7958 7369 7341 7341
q19 6361 1401 1492 1401
q20 643 315 323 315
q21 4949 3241 3882 3241
q22 397 335 346 335
Total cold run time: 117285 ms
Total hot run time: 39769 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4524 4315 4244 4244
q2 362 256 268 256
q3 3013 2935 2916 2916
q4 2016 1720 1760 1720
q5 5594 5507 5515 5507
q6 221 128 126 126
q7 2219 1878 1834 1834
q8 3260 3395 3465 3395
q9 8731 8642 8801 8642
q10 4199 3808 3765 3765
q11 616 495 496 495
q12 813 616 637 616
q13 15908 3216 3121 3121
q14 296 279 276 276
q15 528 494 480 480
q16 484 436 461 436
q17 1809 1517 1492 1492
q18 8387 8132 7789 7789
q19 1815 1541 1633 1541
q20 3036 1890 1893 1890
q21 5648 4949 4750 4750
q22 637 552 558 552
Total cold run time: 74116 ms
Total hot run time: 55843 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2173313803
<details>
<summary>ClickBench: <b>Total hot run time: 31.07 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 41d2b015a273f3a2ede855950279317bb883abc3, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.03 0.03
query3 0.22 0.05 0.05
query4 1.68 0.07 0.06
query5 0.50 0.49 0.50
query6 1.12 0.73 0.72
query7 0.02 0.01 0.01
query8 0.04 0.04 0.04
query9 0.54 0.48 0.50
query10 0.54 0.54 0.54
query11 0.14 0.11 0.11
query12 0.14 0.12 0.11
query13 0.59 0.58 0.58
query14 0.76 0.78 0.80
query15 0.82 0.80 0.82
query16 0.36 0.34 0.37
query17 0.94 0.96 0.98
query18 0.23 0.24 0.26
query19 1.76 1.73 1.82
query20 0.01 0.01 0.01
query21 15.41 0.66 0.65
query22 3.95 7.14 2.55
query23 18.33 1.40 1.29
query24 2.14 0.23 0.21
query25 0.16 0.09 0.08
query26 0.26 0.18 0.18
query27 0.08 0.07 0.07
query28 13.18 1.00 0.99
query29 12.65 3.23 3.18
query30 0.26 0.06 0.05
query31 2.85 0.38 0.39
query32 3.30 0.47 0.47
query33 2.89 2.94 2.90
query34 17.00 4.46 4.43
query35 4.66 4.49 4.56
query36 0.63 0.46 0.46
query37 0.18 0.15 0.15
query38 0.16 0.15 0.14
query39 0.04 0.03 0.03
query40 0.18 0.16 0.13
query41 0.10 0.04 0.05
query42 0.05 0.05 0.04
query43 0.04 0.04 0.04
Total cold run time: 109.03 s
Total hot run time: 31.07 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #35875:
URL: https://github.com/apache/doris/pull/35875#discussion_r1635870616
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -745,8 +747,32 @@ private ColumnStatistic getColumnStatistic(TableIf table, String colName, long i
return ColumnStatistic.UNKNOWN;
}
} else {
+ if (!partitionNames.isEmpty()) {
+ PartitionColumnStatisticBuilder builder = new PartitionColumnStatisticBuilder();
+ boolean isFirst = true;
+ boolean hasUnknown = false;
+ for (String partitionName : partitionNames) {
+ PartitionColumnStatistic pcolStats = Env.getCurrentEnv().getStatisticsCache()
+ .getPartitionColumnStatistics(
+ catalogId, dbId, table.getId(), idxId, partitionName, colName);
+ if (pcolStats.isUnKnown) {
+ hasUnknown = true;
+ break;
+ }
+ if (isFirst) {
+ builder = new PartitionColumnStatisticBuilder(pcolStats);
+ isFirst = false;
+ } else {
+ builder.merge(pcolStats);
Review Comment:
do merge after check all pcol stats is not unknown to avoid useless merge?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -764,19 +790,22 @@ private Statistics computeCatalogRelation(CatalogRelation catalogRelation) {
Set<SlotReference> slotSet = slotSetBuilder.build();
Map<Expression, ColumnStatisticBuilder> columnStatisticBuilderMap = new HashMap<>();
TableIf table = catalogRelation.getTable();
- boolean isOlapTable = table instanceof OlapTable;
AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
TableStatsMeta tableMeta = analysisManager.findTableStatsStatus(table.getId());
long tableUpdatedRows = tableMeta == null ? 0 : tableMeta.updatedRows.get();
- double rowCount = catalogRelation.getTable().getRowCountForNereids();
boolean hasUnknownKeyCol = false;
long idxId = -1;
+ List<String> selectedPartitionNames = new ArrayList<>();
Review Comment:
for better perf, use expected size list to avoid resize
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatisticBuilder.java:
##########
@@ -155,4 +155,25 @@ public PartitionColumnStatistic build() {
dataSize, minValue, maxValue, minExpr, maxExpr,
isUnknown, updatedTime);
}
+
+ public PartitionColumnStatisticBuilder merge(PartitionColumnStatistic other) {
+ count += other.count;
Review Comment:
```suggestion
count += other.count;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2174737382
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2164804883
add description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) support partition level column stats [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2185439661
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #35875:
URL: https://github.com/apache/doris/pull/35875#discussion_r1636042617
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/PartitionColumnStatisticBuilder.java:
##########
@@ -155,4 +155,25 @@ public PartitionColumnStatistic build() {
dataSize, minValue, maxValue, minExpr, maxExpr,
isUnknown, updatedTime);
}
+
+ public PartitionColumnStatisticBuilder merge(PartitionColumnStatistic other) {
+ count += other.count;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2162522518
<details>
<summary>TPC-DS: <b>Total hot run time: 174375 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0cf12548e96986bab337a63ef39cf43c448a7e30, data reload: false
query1 930 377 383 377
query2 6394 2570 2467 2467
query3 6625 207 207 207
query4 18917 17320 17224 17224
query5 3515 485 476 476
query6 235 157 175 157
query7 4577 287 296 287
query8 309 293 273 273
query9 8581 2405 2354 2354
query10 572 317 285 285
query11 10380 10206 10021 10021
query12 122 87 84 84
query13 1653 392 370 370
query14 10178 7980 7868 7868
query15 238 198 185 185
query16 8017 263 269 263
query17 1908 546 531 531
query18 2086 272 270 270
query19 201 153 154 153
query20 90 80 80 80
query21 208 129 132 129
query22 4249 4140 4088 4088
query23 33676 33676 33627 33627
query24 11120 2854 2949 2854
query25 619 384 396 384
query26 1204 163 163 163
query27 2336 333 333 333
query28 6868 2115 2088 2088
query29 905 672 652 652
query30 274 161 162 161
query31 982 776 749 749
query32 94 57 58 57
query33 785 301 341 301
query34 978 503 498 498
query35 772 632 622 622
query36 1121 985 961 961
query37 149 81 79 79
query38 2957 2898 2758 2758
query39 899 864 830 830
query40 207 125 129 125
query41 56 52 58 52
query42 111 102 105 102
query43 610 593 562 562
query44 1251 758 739 739
query45 195 165 163 163
query46 1075 739 730 730
query47 1856 1749 1746 1746
query48 374 301 336 301
query49 860 407 419 407
query50 768 394 400 394
query51 6779 6623 6641 6623
query52 108 91 95 91
query53 364 297 292 292
query54 871 450 439 439
query55 75 74 73 73
query56 277 262 264 262
query57 1139 1040 1049 1040
query58 257 279 258 258
query59 3394 3093 3088 3088
query60 293 275 283 275
query61 95 92 89 89
query62 612 464 439 439
query63 321 287 295 287
query64 8719 2266 1733 1733
query65 3139 3267 3110 3110
query66 781 327 337 327
query67 15301 14869 14956 14869
query68 4484 554 559 554
query69 455 322 323 322
query70 1173 1165 1172 1165
query71 396 273 281 273
query72 7250 5842 5075 5075
query73 743 329 327 327
query74 5954 5505 5438 5438
query75 3379 2656 2691 2656
query76 2222 915 906 906
query77 443 306 297 297
query78 10548 9838 9785 9785
query79 2588 506 519 506
query80 2075 456 453 453
query81 569 221 229 221
query82 951 100 101 100
query83 273 169 169 169
query84 244 92 82 82
query85 1958 266 263 263
query86 518 325 313 313
query87 3267 3059 3078 3059
query88 4238 2454 2455 2454
query89 475 371 374 371
query90 1710 183 188 183
query91 129 95 99 95
query92 61 47 48 47
query93 3126 506 497 497
query94 1101 188 185 185
query95 402 320 309 309
query96 604 278 270 270
query97 3201 3082 3085 3082
query98 218 207 197 197
query99 1140 848 839 839
Total cold run time: 271507 ms
Total hot run time: 174375 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #35875:
URL: https://github.com/apache/doris/pull/35875#discussion_r1636183048
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -745,8 +747,32 @@ private ColumnStatistic getColumnStatistic(TableIf table, String colName, long i
return ColumnStatistic.UNKNOWN;
}
} else {
+ if (!partitionNames.isEmpty()) {
+ PartitionColumnStatisticBuilder builder = new PartitionColumnStatisticBuilder();
+ boolean isFirst = true;
+ boolean hasUnknown = false;
+ for (String partitionName : partitionNames) {
+ PartitionColumnStatistic pcolStats = Env.getCurrentEnv().getStatisticsCache()
+ .getPartitionColumnStatistics(
+ catalogId, dbId, table.getId(), idxId, partitionName, colName);
+ if (pcolStats.isUnKnown) {
+ hasUnknown = true;
+ break;
+ }
+ if (isFirst) {
+ builder = new PartitionColumnStatisticBuilder(pcolStats);
+ isFirst = false;
+ } else {
+ builder.merge(pcolStats);
Review Comment:
delay merge after check all partition could get beter perf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) support partition level column stats [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2185439625
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #35875:
URL: https://github.com/apache/doris/pull/35875#discussion_r1636030225
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java:
##########
@@ -745,8 +747,32 @@ private ColumnStatistic getColumnStatistic(TableIf table, String colName, long i
return ColumnStatistic.UNKNOWN;
}
} else {
+ if (!partitionNames.isEmpty()) {
+ PartitionColumnStatisticBuilder builder = new PartitionColumnStatisticBuilder();
+ boolean isFirst = true;
+ boolean hasUnknown = false;
+ for (String partitionName : partitionNames) {
+ PartitionColumnStatistic pcolStats = Env.getCurrentEnv().getStatisticsCache()
+ .getPartitionColumnStatistics(
+ catalogId, dbId, table.getId(), idxId, partitionName, colName);
+ if (pcolStats.isUnKnown) {
+ hasUnknown = true;
+ break;
+ }
+ if (isFirst) {
+ builder = new PartitionColumnStatisticBuilder(pcolStats);
+ isFirst = false;
+ } else {
+ builder.merge(pcolStats);
Review Comment:
this check is done in line758
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2173179793
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2173279125
<details>
<summary>TPC-H: <b>Total hot run time: 39630 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 41d2b015a273f3a2ede855950279317bb883abc3, data reload: false
------ Round 1 ----------------------------------
q1 17595 4292 4240 4240
q2 2008 185 186 185
q3 10474 1089 1139 1089
q4 10191 830 804 804
q5 7476 2647 2595 2595
q6 212 128 129 128
q7 932 598 583 583
q8 9220 2055 2066 2055
q9 9016 6481 6450 6450
q10 8949 3736 3765 3736
q11 460 234 230 230
q12 475 227 219 219
q13 17778 2976 3013 2976
q14 268 216 216 216
q15 512 477 476 476
q16 500 383 386 383
q17 954 708 663 663
q18 7975 7439 7379 7379
q19 7470 1459 1459 1459
q20 656 314 312 312
q21 4891 3119 3914 3119
q22 380 333 340 333
Total cold run time: 118392 ms
Total hot run time: 39630 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4365 4285 4199 4199
q2 382 269 267 267
q3 3018 2893 2896 2893
q4 2028 1675 1801 1675
q5 5583 5495 5458 5458
q6 214 134 124 124
q7 2266 1885 1849 1849
q8 3269 3439 3401 3401
q9 8689 8838 8762 8762
q10 4058 3814 3729 3729
q11 595 499 507 499
q12 830 633 630 630
q13 17003 3170 3211 3170
q14 300 264 280 264
q15 518 484 498 484
q16 484 425 427 425
q17 1811 1512 1504 1504
q18 8112 7997 7825 7825
q19 1862 1661 1633 1633
q20 3102 1916 1843 1843
q21 5084 4896 4833 4833
q22 635 529 543 529
Total cold run time: 74208 ms
Total hot run time: 55996 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2177415298
<details>
<summary>TPC-DS: <b>Total hot run time: 173420 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 69f32fa800535c3026114840f838214f603a9b58, data reload: false
query1 923 378 389 378
query2 6431 2245 2370 2245
query3 6631 205 197 197
query4 19250 17311 17287 17287
query5 3556 460 472 460
query6 242 166 164 164
query7 4585 291 300 291
query8 310 265 266 265
query9 8451 2382 2382 2382
query10 541 288 262 262
query11 10471 10205 10043 10043
query12 111 85 80 80
query13 1638 354 372 354
query14 9321 7682 6649 6649
query15 214 180 184 180
query16 7741 273 271 271
query17 1731 561 532 532
query18 1954 287 278 278
query19 195 166 161 161
query20 88 83 81 81
query21 203 126 125 125
query22 4356 4234 4106 4106
query23 33873 33596 33594 33594
query24 11092 2890 2922 2890
query25 640 406 380 380
query26 1171 155 158 155
query27 2389 322 327 322
query28 7322 2123 2097 2097
query29 909 659 633 633
query30 257 156 163 156
query31 972 780 775 775
query32 96 56 58 56
query33 788 303 303 303
query34 985 477 494 477
query35 769 662 622 622
query36 1149 971 968 968
query37 162 75 77 75
query38 3002 2884 2866 2866
query39 886 845 838 838
query40 215 136 128 128
query41 54 53 55 53
query42 109 102 105 102
query43 577 535 531 531
query44 1194 726 718 718
query45 197 164 164 164
query46 1082 720 742 720
query47 1914 1818 1785 1785
query48 370 315 306 306
query49 867 414 430 414
query50 767 382 385 382
query51 6811 6707 6736 6707
query52 106 99 94 94
query53 359 285 301 285
query54 935 442 442 442
query55 76 72 74 72
query56 299 362 252 252
query57 1101 1058 1052 1052
query58 258 250 230 230
query59 3276 3068 3109 3068
query60 298 267 265 265
query61 89 88 112 88
query62 593 471 430 430
query63 316 288 281 281
query64 8776 2203 1747 1747
query65 3157 3073 3080 3073
query66 865 320 322 320
query67 15818 15327 15043 15043
query68 8907 543 552 543
query69 720 453 413 413
query70 1395 1163 1138 1138
query71 539 276 280 276
query72 8740 5823 5387 5387
query73 2278 327 320 320
query74 5888 5570 5518 5518
query75 5816 2613 2612 2612
query76 5571 971 947 947
query77 786 291 289 289
query78 10432 10032 9752 9752
query79 7702 519 515 515
query80 2229 465 459 459
query81 564 217 229 217
query82 265 96 105 96
query83 299 163 164 163
query84 266 83 85 83
query85 926 319 265 265
query86 346 296 317 296
query87 3286 3075 3039 3039
query88 4402 2334 2330 2330
query89 525 375 381 375
query90 1950 185 187 185
query91 124 95 97 95
query92 59 50 48 48
query93 5822 501 506 501
query94 1275 185 184 184
query95 397 312 307 307
query96 588 268 264 264
query97 3246 3085 3051 3051
query98 227 196 192 192
query99 1292 810 824 810
Total cold run time: 292746 ms
Total hot run time: 173420 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2160554946
<details>
<summary>ClickBench: <b>Total hot run time: 29.85 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d101cfd0875a21af3df33ce7b1b1d0103088825e, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.04 0.04
query3 0.23 0.05 0.06
query4 1.68 0.08 0.08
query5 0.49 0.48 0.50
query6 1.12 0.73 0.73
query7 0.02 0.01 0.02
query8 0.04 0.04 0.04
query9 0.53 0.50 0.50
query10 0.53 0.54 0.54
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.61 0.60 0.58
query14 0.78 0.79 0.80
query15 0.83 0.80 0.81
query16 0.36 0.35 0.37
query17 0.94 1.01 0.98
query18 0.21 0.22 0.26
query19 1.80 1.83 1.68
query20 0.01 0.01 0.00
query21 15.40 0.65 0.65
query22 3.79 8.75 1.44
query23 18.27 1.22 1.29
query24 2.09 0.25 0.21
query25 0.15 0.08 0.08
query26 0.26 0.17 0.17
query27 0.08 0.08 0.09
query28 13.27 1.02 1.00
query29 12.56 3.27 3.24
query30 0.25 0.06 0.05
query31 2.88 0.38 0.38
query32 3.29 0.47 0.46
query33 2.90 2.86 2.94
query34 17.08 4.37 4.45
query35 4.43 4.44 4.44
query36 0.66 0.47 0.48
query37 0.19 0.15 0.15
query38 0.14 0.13 0.14
query39 0.04 0.04 0.04
query40 0.17 0.14 0.14
query41 0.10 0.04 0.04
query42 0.06 0.05 0.05
query43 0.04 0.04 0.04
Total cold run time: 108.7 s
Total hot run time: 29.85 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2160031809
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2160508609
<details>
<summary>TPC-H: <b>Total hot run time: 39649 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d101cfd0875a21af3df33ce7b1b1d0103088825e, data reload: false
------ Round 1 ----------------------------------
q1 18053 4535 4364 4364
q2 2658 192 197 192
q3 11442 1100 1154 1100
q4 11072 788 746 746
q5 8584 2694 2608 2608
q6 221 134 146 134
q7 941 588 614 588
q8 9224 2057 2058 2057
q9 8876 6483 6497 6483
q10 8985 3694 3715 3694
q11 449 230 225 225
q12 479 225 213 213
q13 17770 2991 2966 2966
q14 257 218 210 210
q15 500 461 473 461
q16 495 387 370 370
q17 947 603 686 603
q18 8033 7534 7338 7338
q19 6766 1459 1490 1459
q20 646 303 308 303
q21 4865 3207 3849 3207
q22 384 345 328 328
Total cold run time: 121647 ms
Total hot run time: 39649 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4483 4237 4254 4237
q2 380 279 277 277
q3 3006 2735 2786 2735
q4 1853 1561 1561 1561
q5 5237 5280 5258 5258
q6 220 127 127 127
q7 2142 1741 1734 1734
q8 3218 3331 3298 3298
q9 8328 8338 8311 8311
q10 3819 3614 3627 3614
q11 589 503 487 487
q12 797 593 578 578
q13 17432 3001 3006 3001
q14 292 282 256 256
q15 509 482 490 482
q16 480 423 415 415
q17 1762 1474 1494 1474
q18 7706 7652 7318 7318
q19 2713 1669 1507 1507
q20 1976 1775 1792 1775
q21 5018 4715 4642 4642
q22 621 512 534 512
Total cold run time: 72581 ms
Total hot run time: 53599 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2177354451
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids)support partition level column stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2177421086
<details>
<summary>ClickBench: <b>Total hot run time: 30.69 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 69f32fa800535c3026114840f838214f603a9b58, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.05
query3 0.22 0.05 0.05
query4 1.68 0.07 0.07
query5 0.50 0.49 0.51
query6 1.12 0.75 0.73
query7 0.02 0.01 0.01
query8 0.05 0.04 0.03
query9 0.55 0.48 0.49
query10 0.52 0.53 0.53
query11 0.16 0.12 0.12
query12 0.16 0.12 0.12
query13 0.61 0.59 0.59
query14 0.77 0.81 0.82
query15 0.86 0.84 0.84
query16 0.38 0.36 0.38
query17 0.96 0.99 1.00
query18 0.22 0.25 0.22
query19 1.80 1.70 1.73
query20 0.01 0.01 0.01
query21 15.40 0.65 0.64
query22 4.99 6.72 1.87
query23 18.28 1.40 1.25
query24 2.07 0.24 0.21
query25 0.15 0.08 0.08
query26 0.27 0.17 0.18
query27 0.08 0.07 0.09
query28 13.28 1.05 1.03
query29 12.62 3.33 3.31
query30 0.25 0.06 0.06
query31 2.86 0.41 0.40
query32 3.26 0.50 0.48
query33 2.93 2.96 2.88
query34 17.26 4.48 4.44
query35 4.52 4.53 4.63
query36 0.65 0.48 0.47
query37 0.18 0.15 0.16
query38 0.14 0.14 0.13
query39 0.04 0.04 0.04
query40 0.19 0.15 0.14
query41 0.09 0.04 0.04
query42 0.05 0.04 0.04
query43 0.05 0.04 0.04
Total cold run time: 110.32 s
Total hot run time: 30.69 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] merge part stats [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35875:
URL: https://github.com/apache/doris/pull/35875#issuecomment-2147336187
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org