You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2026/03/12 08:18:54 UTC
[PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Mryange opened a new pull request, #61260:
URL: https://github.com/apache/doris/pull/61260
### What problem does this PR solve?
For SQL like:
```mysql
SELECT xxx, count(*) FROM table GROUP BY ...
```
we can apply the following optimization:
In the agg hashmap<key, value>, the value is a char*, which happens to be 64-bit. We can treat this pointer directly as a uint64 counter. This avoids creating an AggState. However, this introduces extra if/else branches in many places where we operate on the hashmap. We plan to refactor this area in the future.
```mysql
MySQL [hits]> SELECT ClientIP,sum(ClientIP) AS c
-> FROM hits
-> GROUP BY ClientIP
-> ORDER BY c DESC
-> LIMIT 10;
10 rows in set (0.374 sec)
MySQL [hits]> SELECT ClientIP, COUNT(*) AS c
-> FROM hits
-> GROUP BY ClientIP
-> ORDER BY c DESC
-> LIMIT 10;
10 rows in set (0.312 sec)
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4063203774
<details>
<summary>TPC-H: <b>Total hot run time: 26408 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce542e06a38f506a34a230ada69fb06d88cca94e, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17678 4526 4288 4288
q2 q3 10632 847 523 523
q4 4678 358 251 251
q5 7551 1203 1014 1014
q6 174 174 146 146
q7 775 846 670 670
q8 9305 1469 1308 1308
q9 4888 4807 4698 4698
q10 6255 1920 1660 1660
q11 459 259 242 242
q12 703 575 479 479
q13 18032 2738 1923 1923
q14 228 243 214 214
q15 q16 740 725 668 668
q17 708 851 466 466
q18 6041 5468 5183 5183
q19 1149 993 623 623
q20 538 486 379 379
q21 4379 1824 1387 1387
q22 333 286 319 286
Total cold run time: 95246 ms
Total hot run time: 26408 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4887 4633 4588 4588
q2 q3 3867 4339 3818 3818
q4 875 1184 774 774
q5 4090 4397 4321 4321
q6 176 173 140 140
q7 1772 1646 1521 1521
q8 2491 2722 2593 2593
q9 7632 7303 7509 7303
q10 3768 4149 3602 3602
q11 532 439 437 437
q12 498 575 452 452
q13 2461 2818 1970 1970
q14 285 288 278 278
q15 q16 719 840 701 701
q17 1158 1291 1594 1291
q18 7070 6817 6541 6541
q19 865 866 858 858
q20 2089 2129 1974 1974
q21 3998 3448 3427 3427
q22 441 431 395 395
Total cold run time: 49674 ms
Total hot run time: 46984 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4102333348
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1a9a1b495f832915dd6b13a13eaaccfdeced2bf5_1a9a1b495f832915dd6b13a13eaaccfdeced2bf5/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1a9a1b495f832915dd6b13a13eaaccfdeced2bf5_1a9a1b495f832915dd6b13a13eaaccfdeced2bf5/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.76% (19847/37615) |
| Line Coverage | 36.26% (185369/511188) |
| Region Coverage | 32.52% (143502/441279) |
| Branch Coverage | 33.72% (62825/186305) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4087753827
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4063407387
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61260_ce542e06a38f506a34a230ada69fb06d88cca94e_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61260_ce542e06a38f506a34a230ada69fb06d88cca94e_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.55% (26246/36684) |
| Line Coverage | 54.41% (276040/507324) |
| Region Coverage | 51.61% (228940/443619) |
| Branch Coverage | 53.07% (98621/185835) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61260:
URL: https://github.com/apache/doris/pull/61260#discussion_r2964634011
##########
be/src/exec/operator/aggregation_sink_operator.cpp:
##########
@@ -25,6 +25,7 @@
#include "core/data_type/primitive_type.h"
#include "exec/common/hash_table/hash.h"
#include "exec/operator/operator.h"
+#include "exprs/aggregate/aggregate_function_count.h"
#include "exprs/aggregate/aggregate_function_simple_factory.h"
#include "exprs/vectorized_agg_fn.h"
#include "runtime/runtime_profile.h"
Review Comment:
**Minor: Comment says "count(*) / count(not_null_column)" but only count(*) is actually supported**
`AggregateFunctionCountNotNullUnary` does NOT override `is_simple_count()` (it stays `false`). So this optimization only applies to `AggregateFunctionCount` (the `count(*)` variant), not to `count(not_null_column)`. The comment is misleading.
Also, consider adding `static_assert(sizeof(AggregateDataPtr) == sizeof(UInt64))` somewhere in this file to document and enforce the fundamental 64-bit assumption of this optimization.
##########
be/src/exec/operator/streaming_aggregation_operator.cpp:
##########
@@ -97,22 +99,36 @@ Status StreamingAggLocalState::open(RuntimeState* state) {
RETURN_IF_ERROR(_init_hash_method(_probe_expr_ctxs));
- std::visit(Overload {[&](std::monostate& arg) -> void {
- throw doris::Exception(ErrorCode::INTERNAL_ERROR,
- "uninited hash table");
- },
Review Comment:
**Bug: `limit == -1` check is always true (tautology)**
At this point, `limit` is still its default value `-1` (initialized in the header at line 106). The actual assignment `limit = p._sort_limit` happens later at line 133.
This means when `p._sort_limit > 0` (sort-limit optimization active), `_use_simple_count` will incorrectly be set to `true`, and `_aggregate_data_container` will be `nullptr` (skipped at line 122). This leads to a null pointer dereference when `build_limit_heap()` → `_get_keys_hash_table()` accesses `_aggregate_data_container->begin()`.
**Fix:** Either move `limit = p._sort_limit` before this check, or use `p._sort_limit` directly:
```cpp
if (_aggregate_evaluators.size() == 1 &&
_aggregate_evaluators[0]->function()->is_simple_count() && p._sort_limit == -1) {
```
Note: The analogous check in `AggSinkLocalState::open()` (aggregation_sink_operator.cpp:175) correctly uses `!_should_limit_output` which is already set by that point.
##########
be/src/exec/pipeline/dependency.h:
##########
@@ -320,6 +320,7 @@ struct AggSharedState : public BasicSharedState {
bool enable_spill = false;
bool reach_limit = false;
+ bool use_simple_count = false;
int64_t limit = -1;
bool do_sort_limit = false;
MutableColumns limit_columns;
Review Comment:
**Latent bug: `reset_hash_table()` in `dependency.cpp:278-311` lacks `use_simple_count` guard**
`_close_with_serialized_key()` here correctly checks `use_simple_count` before calling `_destroy_agg_status()`. However, `AggSharedState::reset_hash_table()` (dependency.cpp:290-295) does NOT check `use_simple_count` before calling `_destroy_agg_status()` on mapped values.
Currently safe because `use_simple_count` requires `!enable_spill` and `reset_hash_table()` is only called from spill paths. But if the spill exclusion is ever relaxed, this becomes a wild-pointer dereference crash.
Suggestion: Add a matching guard or at minimum a `DCHECK(!use_simple_count)` in `reset_hash_table()` for safety.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4048801017
<details>
<summary>TPC-DS: <b>Total hot run time: 154734 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8497d2aa28aad9c1db55c8b24e70401e448c02a3, data reload: false
query5 4341 626 499 499
query6 332 227 215 215
query7 4204 465 269 269
query8 351 249 235 235
query9 8701 2771 2735 2735
query10 542 397 334 334
query11 7408 5888 5655 5655
query12 187 128 127 127
query13 1251 442 338 338
query14 6009 3814 3621 3621
query14_1 2792 2791 2797 2791
query15 205 194 176 176
query16 1009 469 457 457
query17 1078 698 598 598
query18 2432 437 337 337
query19 208 215 178 178
query20 142 133 131 131
query21 222 137 120 120
query22 4841 4904 4936 4904
query23 17129 16472 16743 16472
query23_1 16213 16424 16328 16328
query24 7275 1597 1249 1249
query24_1 1254 1196 1242 1196
query25 524 456 412 412
query26 1244 250 149 149
query27 2800 481 293 293
query28 4515 1864 1862 1862
query29 855 560 478 478
query30 329 247 201 201
query31 1360 1284 1235 1235
query32 88 68 71 68
query33 502 325 275 275
query34 935 887 575 575
query35 622 668 600 600
query36 1076 1152 958 958
query37 134 95 82 82
query38 2907 2943 2907 2907
query39 871 872 846 846
query39_1 841 830 822 822
query40 231 154 135 135
query41 65 62 58 58
query42 308 302 297 297
query43 239 254 215 215
query44
query45 201 189 184 184
query46 871 976 602 602
query47 2139 2156 2046 2046
query48 308 307 229 229
query49 636 460 383 383
query50 676 282 209 209
query51 4121 4122 4094 4094
query52 280 302 287 287
query53 288 333 292 292
query54 300 271 277 271
query55 99 89 80 80
query56 316 323 319 319
query57 1386 1346 1304 1304
query58 294 279 282 279
query59 1313 1491 1271 1271
query60 333 340 334 334
query61 149 152 145 145
query62 641 577 552 552
query63 312 276 276 276
query64 4988 1326 1120 1120
query65
query66 1483 485 373 373
query67 16228 16347 16224 16224
query68
query69 405 320 296 296
query70 1009 954 986 954
query71 333 314 302 302
query72 2997 2840 2692 2692
query73 543 554 322 322
query74 9969 9967 9771 9771
query75 2822 2761 2438 2438
query76 2272 1029 659 659
query77 370 384 303 303
query78 11198 11318 10655 10655
query79 3075 807 606 606
query80 1729 611 532 532
query81 586 287 244 244
query82 977 150 119 119
query83 329 277 239 239
query84 258 117 103 103
query85 894 491 451 451
query86 497 313 303 303
query87 3124 3118 2993 2993
query88 3510 2664 2649 2649
query89 419 375 347 347
query90 2045 177 166 166
query91 164 155 136 136
query92 84 75 71 71
query93 1539 830 495 495
query94 637 353 306 306
query95 596 406 318 318
query96 640 513 230 230
query97 2489 2499 2472 2472
query98 242 228 222 222
query99 969 995 914 914
Total cold run time: 237783 ms
Total hot run time: 154734 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4063397966
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ce542e06a38f506a34a230ada69fb06d88cca94e_ce542e06a38f506a34a230ada69fb06d88cca94e/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ce542e06a38f506a34a230ada69fb06d88cca94e_ce542e06a38f506a34a230ada69fb06d88cca94e/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.61% (19709/37465) |
| Line Coverage | 36.20% (184230/508881) |
| Region Coverage | 32.30% (141959/439482) |
| Branch Coverage | 33.53% (62121/185273) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4102121625
<details>
<summary>TPC-DS: <b>Total hot run time: 169609 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1a9a1b495f832915dd6b13a13eaaccfdeced2bf5, data reload: false
query5 4352 636 521 521
query6 355 240 209 209
query7 4224 486 272 272
query8 343 251 233 233
query9 8765 2755 2729 2729
query10 490 394 375 375
query11 7011 5085 4886 4886
query12 192 136 127 127
query13 1290 465 363 363
query14 5769 3754 3471 3471
query14_1 2847 2811 2853 2811
query15 201 195 178 178
query16 998 483 466 466
query17 898 758 641 641
query18 2457 461 355 355
query19 225 215 199 199
query20 138 132 130 130
query21 218 135 113 113
query22 13348 14163 15228 14163
query23 16841 16248 15971 15971
query23_1 16353 16197 15977 15977
query24 7144 1609 1251 1251
query24_1 1214 1230 1224 1224
query25 529 467 411 411
query26 1233 266 147 147
query27 2743 486 311 311
query28 4425 1854 1848 1848
query29 806 569 474 474
query30 299 224 191 191
query31 1023 953 865 865
query32 87 68 72 68
query33 498 343 323 323
query34 892 875 517 517
query35 634 711 610 610
query36 1074 1101 985 985
query37 145 95 82 82
query38 2958 2929 2945 2929
query39 865 835 825 825
query39_1 793 787 803 787
query40 235 156 133 133
query41 64 60 58 58
query42 260 254 265 254
query43 254 239 217 217
query44
query45 202 198 188 188
query46 872 986 618 618
query47 2118 2897 2079 2079
query48 314 320 236 236
query49 636 468 380 380
query50 719 272 224 224
query51 4053 4096 4063 4063
query52 266 271 263 263
query53 290 346 288 288
query54 305 287 275 275
query55 92 85 78 78
query56 335 338 300 300
query57 1924 1670 1525 1525
query58 290 277 272 272
query59 2799 2986 2745 2745
query60 342 334 330 330
query61 153 156 152 152
query62 643 587 550 550
query63 328 289 276 276
query64 4958 1318 1031 1031
query65
query66 1460 451 361 361
query67 24336 24375 24268 24268
query68
query69 411 315 289 289
query70 985 903 982 903
query71 337 312 301 301
query72 2798 2673 2388 2388
query73 553 547 315 315
query74 9667 9634 9434 9434
query75 2889 2761 2479 2479
query76 2268 1045 676 676
query77 366 410 311 311
query78 10916 11190 10502 10502
query79 1157 778 574 574
query80 1056 640 540 540
query81 524 259 224 224
query82 1341 159 124 124
query83 359 269 276 269
query84 294 123 100 100
query85 910 507 451 451
query86 415 305 304 304
query87 3179 3204 2983 2983
query88 3592 2667 2649 2649
query89 426 370 346 346
query90 1889 184 187 184
query91 168 159 140 140
query92 79 82 73 73
query93 921 862 497 497
query94 509 333 290 290
query95 596 342 390 342
query96 657 538 229 229
query97 2479 2486 2420 2420
query98 236 222 217 217
query99 1005 1016 924 924
Total cold run time: 249811 ms
Total hot run time: 169609 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4108297804
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4102377067
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61260_1a9a1b495f832915dd6b13a13eaaccfdeced2bf5_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61260_1a9a1b495f832915dd6b13a13eaaccfdeced2bf5_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.26% (26982/36831) |
| Line Coverage | 56.77% (289311/509634) |
| Region Coverage | 54.15% (241197/445418) |
| Branch Coverage | 55.81% (104299/186873) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #61260:
URL: https://github.com/apache/doris/pull/61260#discussion_r2960893504
##########
be/src/exec/operator/streaming_aggregation_operator.cpp:
##########
@@ -456,6 +480,74 @@ Status StreamingAggLocalState::_get_results_with_serialized_key(RuntimeState* st
const auto size = std::min(data.size(), size_t(state->batch_size()));
using KeyType = std::decay_t<decltype(agg_method)>::Key;
std::vector<KeyType> keys(size);
+
+ if (_use_simple_count) {
+ DCHECK_EQ(_aggregate_evaluators.size(), 1);
+
+ value_data_types[0] =
+ _aggregate_evaluators[0]->function()->get_serialized_type();
+ if (mem_reuse) {
+ value_columns[0] =
+ std::move(*block->get_by_position(key_size).column)
+ .mutate();
+ } else {
+ value_columns[0] = _aggregate_evaluators[0]
+ ->function()
+ ->create_serialize_column();
+ }
+
+ std::vector<UInt64> inline_counts(size);
Review Comment:
why here do not directly use `auto& count_col =
assert_cast<ColumnFixedLengthObject&>(*value_columns[0]);
count_col.resize(num_rows);` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4046144310
# BE UT Coverage Report
Increment line coverage `14.06% (45/320)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/8497d2aa28aad9c1db55c8b24e70401e448c02a3_8497d2aa28aad9c1db55c8b24e70401e448c02a3/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/8497d2aa28aad9c1db55c8b24e70401e448c02a3_8497d2aa28aad9c1db55c8b24e70401e448c02a3/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.60% (19680/37414) |
| Line Coverage | 36.17% (183664/507793) |
| Region Coverage | 32.27% (141658/438958) |
| Branch Coverage | 33.45% (61861/184919) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4101803877
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #61260:
URL: https://github.com/apache/doris/pull/61260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4044862319
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4047530851
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61260_8497d2aa28aad9c1db55c8b24e70401e448c02a3_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61260_8497d2aa28aad9c1db55c8b24e70401e448c02a3_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.52% (26220/36659) |
| Line Coverage | 54.27% (275168/507045) |
| Region Coverage | 51.21% (227209/443642) |
| Branch Coverage | 52.80% (98083/185763) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4063222214
<details>
<summary>TPC-DS: <b>Total hot run time: 169263 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce542e06a38f506a34a230ada69fb06d88cca94e, data reload: false
query5 4341 644 505 505
query6 339 227 205 205
query7 4224 468 264 264
query8 331 247 224 224
query9 8690 2708 2704 2704
query10 494 395 349 349
query11 6982 5089 4860 4860
query12 185 137 126 126
query13 1262 448 338 338
query14 5762 3677 3449 3449
query14_1 2863 2781 2804 2781
query15 207 197 174 174
query16 981 477 449 449
query17 908 733 611 611
query18 2454 455 352 352
query19 214 209 182 182
query20 132 133 134 133
query21 216 144 113 113
query22 13236 13968 14803 13968
query23 16817 16205 16147 16147
query23_1 16081 16312 16020 16020
query24 7694 1665 1256 1256
query24_1 1316 1330 1256 1256
query25 563 492 437 437
query26 1292 273 163 163
query27 3562 507 334 334
query28 4608 1854 1848 1848
query29 849 545 459 459
query30 297 223 189 189
query31 1014 943 878 878
query32 80 67 73 67
query33 498 332 277 277
query34 885 860 507 507
query35 627 677 594 594
query36 1087 1165 1028 1028
query37 131 91 82 82
query38 2900 2903 2913 2903
query39 857 818 813 813
query39_1 789 777 800 777
query40 231 154 133 133
query41 61 58 57 57
query42 255 263 253 253
query43 250 247 217 217
query44
query45 200 187 182 182
query46 866 981 609 609
query47 2120 2194 2066 2066
query48 307 303 233 233
query49 634 456 380 380
query50 684 267 211 211
query51 4080 4032 4095 4032
query52 267 262 256 256
query53 295 335 285 285
query54 298 284 258 258
query55 87 85 84 84
query56 301 310 314 310
query57 1949 1848 1734 1734
query58 284 269 271 269
query59 2810 2921 2733 2733
query60 343 347 329 329
query61 147 143 145 143
query62 634 593 527 527
query63 305 274 280 274
query64 4973 1273 1009 1009
query65
query66 1461 451 369 369
query67 24252 24322 24185 24185
query68
query69 410 297 278 278
query70 965 953 964 953
query71 339 305 307 305
query72 2769 2649 2407 2407
query73 531 536 319 319
query74 9652 9542 9390 9390
query75 2852 2749 2425 2425
query76 2263 1020 654 654
query77 369 371 308 308
query78 10860 11079 10497 10497
query79 1105 844 556 556
query80 947 603 540 540
query81 521 262 226 226
query82 1373 148 118 118
query83 367 262 240 240
query84 296 111 103 103
query85 872 509 492 492
query86 418 341 304 304
query87 3140 3133 3072 3072
query88 3548 2671 2664 2664
query89 428 378 349 349
query90 1851 181 176 176
query91 186 181 154 154
query92 81 75 75 75
query93 887 846 515 515
query94 513 332 302 302
query95 615 411 326 326
query96 646 514 229 229
query97 2453 2487 2388 2388
query98 254 225 220 220
query99 1024 989 916 916
Total cold run time: 250708 ms
Total hot run time: 169263 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4108297942
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4096575947
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4063015561
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4102065928
<details>
<summary>TPC-H: <b>Total hot run time: 26760 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1a9a1b495f832915dd6b13a13eaaccfdeced2bf5, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17615 4537 4325 4325
q2 q3 10648 799 547 547
q4 4706 357 250 250
q5 7674 1256 1015 1015
q6 176 173 146 146
q7 807 881 673 673
q8 10032 1516 1340 1340
q9 5804 4738 4828 4738
q10 6331 1969 1635 1635
q11 461 252 240 240
q12 745 588 464 464
q13 18067 2687 1945 1945
q14 221 241 206 206
q15 q16 744 737 675 675
q17 756 862 439 439
q18 6030 5398 5227 5227
q19 1507 979 607 607
q20 551 512 385 385
q21 4505 2018 1620 1620
q22 382 349 283 283
Total cold run time: 97762 ms
Total hot run time: 26760 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4803 4633 4658 4633
q2 q3 4036 4385 3840 3840
q4 877 1227 788 788
q5 4052 4440 4329 4329
q6 194 176 144 144
q7 1792 1667 1536 1536
q8 2524 2732 2588 2588
q9 7732 7397 7454 7397
q10 3770 3977 3613 3613
q11 540 456 443 443
q12 507 608 472 472
q13 2609 2930 2098 2098
q14 279 298 290 290
q15 q16 766 778 756 756
q17 1188 1329 1315 1315
q18 7304 6799 6678 6678
q19 938 965 977 965
q20 2138 2254 2054 2054
q21 4063 3511 3458 3458
q22 495 425 395 395
Total cold run time: 50607 ms
Total hot run time: 47792 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #61260:
URL: https://github.com/apache/doris/pull/61260#discussion_r2960893504
##########
be/src/exec/operator/streaming_aggregation_operator.cpp:
##########
@@ -456,6 +480,74 @@ Status StreamingAggLocalState::_get_results_with_serialized_key(RuntimeState* st
const auto size = std::min(data.size(), size_t(state->batch_size()));
using KeyType = std::decay_t<decltype(agg_method)>::Key;
std::vector<KeyType> keys(size);
+
+ if (_use_simple_count) {
+ DCHECK_EQ(_aggregate_evaluators.size(), 1);
+
+ value_data_types[0] =
+ _aggregate_evaluators[0]->function()->get_serialized_type();
+ if (mem_reuse) {
+ value_columns[0] =
+ std::move(*block->get_by_position(key_size).column)
+ .mutate();
+ } else {
+ value_columns[0] = _aggregate_evaluators[0]
+ ->function()
+ ->create_serialize_column();
+ }
+
+ std::vector<UInt64> inline_counts(size);
Review Comment:
why here do not directly use `auto& count_col =
assert_cast<ColumnFixedLengthObject&>(*value_columns[0]);
count_col.resize(num_rows);` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4090754461
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4047949381
run performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4048737460
<details>
<summary>TPC-H: <b>Total hot run time: 27585 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8497d2aa28aad9c1db55c8b24e70401e448c02a3, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17635 4462 4318 4318
q2 q3 10634 819 529 529
q4 4675 362 247 247
q5 7611 1217 1044 1044
q6 188 180 151 151
q7 813 831 671 671
q8 10144 1464 1343 1343
q9 5620 4810 4742 4742
q10 6343 1926 1683 1683
q11 489 268 247 247
q12 737 565 473 473
q13 18037 2686 1955 1955
q14 231 226 222 222
q15 942 808 837 808
q16 737 720 688 688
q17 711 861 435 435
q18 6063 5525 5253 5253
q19 1390 998 589 589
q20 491 490 390 390
q21 4760 1942 1531 1531
q22 400 335 266 266
Total cold run time: 98651 ms
Total hot run time: 27585 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4803 4643 4583 4583
q2 q3 3895 4360 3866 3866
q4 862 1177 780 780
q5 4051 4390 4341 4341
q6 179 178 142 142
q7 1754 1664 1502 1502
q8 2512 2757 2582 2582
q9 7582 7353 7381 7353
q10 3763 4062 3776 3776
q11 527 436 412 412
q12 484 585 432 432
q13 2399 3056 2052 2052
q14 308 314 297 297
q15 847 811 792 792
q16 702 777 723 723
q17 1141 1494 1410 1410
q18 7069 6708 6689 6689
q19 875 851 842 842
q20 2113 2152 2002 2002
q21 3974 3428 3357 3357
q22 494 434 377 377
Total cold run time: 50334 ms
Total hot run time: 48310 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](agg) Optimize the execution of GROUP BY count(*). [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61260:
URL: https://github.com/apache/doris/pull/61260#issuecomment-4044837620
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org