You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "BiteTheDDDDt (via GitHub)" <gi...@apache.org> on 2026/05/22 07:54:51 UTC
[PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
BiteTheDDDDt opened a new pull request, #63529:
URL: https://github.com/apache/doris/pull/63529
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: When `experimental_use_serial_exchange` is enabled and `enable_local_exchange_before_agg` is disabled, a serial exchange source can be followed by a passthrough local exchange before a non-finalizing merge aggregation. For DISTINCT aggregation, that breaks the hash distribution required by the merge aggregation that deduplicates distinct keys. The duplicated keys can then be processed by different local tasks, and later partial sums can produce incorrect results.
This PR preserves hash local shuffle for merge aggregation with a serial child when the aggregation has partition expressions. It also computes the merge flag during `AggSinkOperatorX::init()` because local exchange planning runs before `prepare()`.
### Release note
Fix occasional incorrect DISTINCT aggregate results when serial exchange is enabled.
### Check List (For Author)
- Test: Manual test
- `build-support/clang-format.sh`
- `build-support/check-format.sh`
- `./run-regression-test.sh --run -d nereids_syntax_p0 -s agg_4_phase` (reproduced failure on old BE before patched binary update)
- Behavior changed: Yes. Serial-child merge aggregation now preserves hash local shuffle when required for correctness.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4516674142
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4516886633
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #63529:
URL: https://github.com/apache/doris/pull/63529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4539599989
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4531103220
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4517356212
<details>
<summary>TPC-DS: <b>Total hot run time: 168868 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8e496ea1def13252d8ee1f5793244d1e90efa49, data reload: false
query5 4302 666 515 515
query6 352 209 194 194
query7 4215 571 299 299
query8 336 229 216 216
query9 8869 3967 3940 3940
query10 448 333 325 325
query11 5796 2407 2185 2185
query12 173 128 121 121
query13 1252 590 398 398
query14 5900 5344 4984 4984
query14_1 4334 4336 4282 4282
query15 210 200 180 180
query16 1005 452 431 431
query17 950 703 582 582
query18 2457 490 348 348
query19 228 197 160 160
query20 136 138 129 129
query21 220 142 118 118
query22 13721 13593 13491 13491
query23 17190 16447 16032 16032
query23_1 16084 16094 16136 16094
query24 7555 1749 1300 1300
query24_1 1306 1285 1320 1285
query25 518 459 404 404
query26 1344 318 172 172
query27 2656 554 334 334
query28 4498 1943 1953 1943
query29 1027 622 486 486
query30 299 234 199 199
query31 1111 1071 957 957
query32 88 74 71 71
query33 537 360 290 290
query34 1191 1181 642 642
query35 764 806 673 673
query36 1316 1316 1209 1209
query37 170 108 92 92
query38 3204 3159 3063 3063
query39 930 929 884 884
query39_1 876 879 877 877
query40 230 155 129 129
query41 72 69 69 69
query42 111 114 111 111
query43 322 328 284 284
query44
query45 220 205 204 204
query46 1078 1211 715 715
query47 2333 2344 2155 2155
query48 419 405 313 313
query49 644 503 402 402
query50 1020 340 251 251
query51 4343 4438 4246 4246
query52 107 104 93 93
query53 266 288 206 206
query54 329 282 266 266
query55 97 95 90 90
query56 326 315 320 315
query57 1401 1413 1281 1281
query58 313 285 276 276
query59 1566 1634 1476 1476
query60 344 331 319 319
query61 177 180 178 178
query62 673 630 558 558
query63 241 203 212 203
query64 2479 847 671 671
query65
query66 1757 499 372 372
query67 30106 29978 29859 29859
query68
query69 462 350 313 313
query70 1062 999 954 954
query71 310 319 252 252
query72 3002 2701 2429 2429
query73 850 749 424 424
query74 5094 4895 4758 4758
query75 2669 2575 2248 2248
query76 2276 1153 734 734
query77 391 406 323 323
query78 12151 12242 11508 11508
query79 1499 1052 750 750
query80 768 543 437 437
query81 471 276 235 235
query82 1404 157 118 118
query83 361 281 243 243
query84 309 141 112 112
query85 921 531 461 461
query86 433 328 304 304
query87 3413 3395 3238 3238
query88 3576 2654 2635 2635
query89 441 380 335 335
query90 1833 183 179 179
query91 180 173 142 142
query92 80 80 75 75
query93 1481 1546 923 923
query94 607 350 341 341
query95 669 393 358 358
query96 1012 779 330 330
query97 2686 2685 2565 2565
query98 229 225 228 225
query99 1114 1105 963 963
Total cold run time: 253312 ms
Total hot run time: 168868 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4541025269
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #63529:
URL: https://github.com/apache/doris/pull/63529#discussion_r3286868325
##########
be/src/exec/operator/aggregation_sink_operator.cpp:
##########
@@ -871,6 +871,7 @@ Status AggSinkOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
tnode.agg_node.__isset.agg_sort_infos ? tnode.agg_node.agg_sort_infos[i] : dummy,
tnode.agg_node.grouping_exprs.empty(), false, &evaluator));
_aggregate_evaluators.push_back(evaluator);
+ _is_merge |= evaluator->is_merge();
}
Review Comment:
`_is_merge` is no longer (re)computed in `_calc_aggregate_evaluators()`, and is now only updated in `init()`. In BE unit tests (`BE_TEST`) there are several cases where a `MockAggsinkOperator` is default-constructed, evaluators are pushed, and `prepare()` is called without calling `init()` first (e.g. `be/test/exec/operator/agg_operator_test.cpp:95-103`). With this change, `_is_merge` can remain false/uninitialized, and `AggSinkLocalState::open()` will pick the wrong executor template (or hit UB), potentially breaking tests and behavior. Consider restoring a deterministic `_is_merge` recomputation in `_calc_aggregate_evaluators()` (e.g. reset then OR over evaluators) or otherwise guarantee `init()` is always invoked before `prepare()` in test paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4540484539
<details>
<summary>TPC-H: <b>Total hot run time: 31754 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 97420ea7db825a94f0d23e5c17aff8af887dbfdb, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17708 4151 4076 4076
q2 q3 10853 1372 828 828
q4 4679 479 352 352
q5 7585 2320 2111 2111
q6 236 175 136 136
q7 959 776 650 650
q8 9356 1650 1767 1650
q9 6151 4981 4955 4955
q10 6443 2239 1894 1894
q11 436 278 243 243
q12 694 442 297 297
q13 18176 3408 2783 2783
q14 270 258 236 236
q15 q16 832 773 709 709
q17 975 948 962 948
q18 6953 5928 5612 5612
q19 1177 1333 1139 1139
q20 546 463 271 271
q21 6006 2732 2546 2546
q22 462 383 318 318
Total cold run time: 100497 ms
Total hot run time: 31754 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4820 4772 4977 4772
q2 q3 4947 5282 4646 4646
q4 2117 2209 1429 1429
q5 4972 4802 4693 4693
q6 231 175 132 132
q7 1921 1691 1547 1547
q8 2403 1953 1955 1953
q9 7404 7571 7449 7449
q10 4750 4663 4241 4241
q11 535 387 355 355
q12 746 753 537 537
q13 3012 3371 2791 2791
q14 278 275 252 252
q15 q16 680 698 614 614
q17 1313 1279 1273 1273
q18 7309 6805 6930 6805
q19 1136 1093 1124 1093
q20 2248 2237 1962 1962
q21 5297 4637 4466 4466
q22 512 450 419 419
Total cold run time: 56631 ms
Total hot run time: 51429 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4532422601
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4541023630
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4516637013
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4516673418
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4517279770
<details>
<summary>TPC-H: <b>Total hot run time: 31252 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8e496ea1def13252d8ee1f5793244d1e90efa49, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17590 3852 3904 3852
q2 q3 10766 1352 810 810
q4 4685 480 349 349
q5 7647 2275 2112 2112
q6 249 175 141 141
q7 949 788 626 626
q8 9350 1634 1620 1620
q9 5151 4941 4879 4879
q10 6399 2083 1790 1790
q11 461 276 248 248
q12 627 432 300 300
q13 18105 3382 2821 2821
q14 263 258 232 232
q15 q16 790 783 707 707
q17 942 914 954 914
q18 6974 5940 5541 5541
q19 1310 1268 1062 1062
q20 513 390 323 323
q21 6447 2883 2587 2587
q22 461 372 338 338
Total cold run time: 99679 ms
Total hot run time: 31252 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4718 4457 4598 4457
q2 q3 4830 5169 4552 4552
q4 2136 2200 1388 1388
q5 4701 4739 4582 4582
q6 232 178 137 137
q7 1815 1890 1564 1564
q8 2336 2079 2061 2061
q9 7715 7683 7166 7166
q10 4450 4377 3969 3969
q11 551 383 343 343
q12 699 712 503 503
q13 2986 3376 2809 2809
q14 268 287 248 248
q15 q16 675 710 599 599
q17 1246 1235 1217 1217
q18 7414 6749 6718 6718
q19 1082 1109 1105 1105
q20 2234 2220 1929 1929
q21 5266 4580 4491 4491
q22 530 464 401 401
Total cold run time: 55884 ms
Total hot run time: 50239 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4532664553
<details>
<summary>TPC-DS: <b>Total hot run time: 172436 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a8e496ea1def13252d8ee1f5793244d1e90efa49, data reload: false
query5 4322 672 526 526
query6 330 216 215 215
query7 4228 605 310 310
query8 317 237 225 225
query9 8837 4104 4101 4101
query10 431 335 301 301
query11 5760 2510 2257 2257
query12 171 124 123 123
query13 1260 615 421 421
query14 6213 5531 5244 5244
query14_1 4552 4554 4496 4496
query15 218 207 182 182
query16 983 471 430 430
query17 1041 754 610 610
query18 2466 503 360 360
query19 216 210 168 168
query20 136 133 135 133
query21 218 137 128 128
query22 13643 13604 13374 13374
query23 17576 16682 16245 16245
query23_1 16488 16399 16324 16324
query24 7549 1801 1329 1329
query24_1 1363 1321 1308 1308
query25 536 481 417 417
query26 1317 337 175 175
query27 2691 567 358 358
query28 4444 2021 2006 2006
query29 974 633 499 499
query30 302 242 201 201
query31 1134 1084 972 972
query32 86 73 72 72
query33 546 354 299 299
query34 1184 1168 688 688
query35 798 809 690 690
query36 1436 1377 1320 1320
query37 148 106 86 86
query38 3228 3175 3113 3113
query39 932 919 886 886
query39_1 891 886 889 886
query40 226 146 127 127
query41 64 63 62 62
query42 114 108 107 107
query43 344 348 297 297
query44
query45 220 203 200 200
query46 1142 1222 746 746
query47 2329 2452 2271 2271
query48 411 431 294 294
query49 633 474 382 382
query50 1019 368 259 259
query51 4470 4414 4335 4335
query52 105 106 94 94
query53 264 283 211 211
query54 311 286 251 251
query55 93 93 100 93
query56 329 308 319 308
query57 1467 1473 1351 1351
query58 294 270 268 268
query59 1613 1735 1491 1491
query60 320 322 300 300
query61 159 159 153 153
query62 688 663 591 591
query63 246 198 203 198
query64 2379 797 635 635
query65
query66 1757 469 355 355
query67 30236 29941 29965 29941
query68
query69 471 341 308 308
query70 1029 991 1014 991
query71 301 276 266 266
query72 3050 2757 2417 2417
query73 868 820 427 427
query74 5145 4956 4808 4808
query75 2691 2618 2301 2301
query76 2307 1170 770 770
query77 412 411 349 349
query78 12292 12414 11890 11890
query79 1434 1046 741 741
query80 666 560 447 447
query81 456 277 248 248
query82 1377 162 119 119
query83 355 287 254 254
query84 302 140 118 118
query85 870 536 442 442
query86 403 335 316 316
query87 3423 3416 3273 3273
query88 3685 2805 2718 2718
query89 440 391 345 345
query90 1902 183 185 183
query91 185 175 143 143
query92 82 81 74 74
query93 1470 1416 880 880
query94 537 341 319 319
query95 683 414 367 367
query96 1074 732 339 339
query97 2766 2760 2665 2665
query98 253 234 232 232
query99 1199 1174 1051 1051
Total cold run time: 255370 ms
Total hot run time: 172436 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4531102873
run beut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4517135189
<details>
<summary>TPC-H: <b>Total hot run time: 31364 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dbaee6e6618cc8ada45cadf083115b27c5292fdd, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17764 4003 3812 3812
q2 q3 10805 1394 803 803
q4 4687 475 342 342
q5 7588 2314 2077 2077
q6 239 169 145 145
q7 941 794 636 636
q8 9425 1803 1617 1617
q9 5097 4939 4906 4906
q10 6398 2093 1810 1810
q11 439 281 243 243
q12 631 426 302 302
q13 18117 3384 2808 2808
q14 261 252 233 233
q15 q16 818 764 710 710
q17 1001 958 908 908
q18 7073 5705 5636 5636
q19 1294 1247 1133 1133
q20 548 518 297 297
q21 6149 2940 2631 2631
q22 474 434 315 315
Total cold run time: 99749 ms
Total hot run time: 31364 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4574 4599 4455 4455
q2 q3 4865 5212 4563 4563
q4 2133 2210 1404 1404
q5 4966 4575 4666 4575
q6 229 179 133 133
q7 1863 1733 1530 1530
q8 2328 2063 2161 2063
q9 7662 7165 7208 7165
q10 4492 4404 3985 3985
q11 521 378 351 351
q12 714 723 507 507
q13 2973 3355 2821 2821
q14 274 270 262 262
q15 q16 678 699 609 609
q17 1255 1239 1240 1239
q18 7206 6849 6653 6653
q19 1092 1076 1092 1076
q20 2221 2210 1932 1932
q21 5285 4601 4499 4499
q22 513 450 425 425
Total cold run time: 55844 ms
Total hot run time: 50247 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4517217311
<details>
<summary>TPC-DS: <b>Total hot run time: 169132 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dbaee6e6618cc8ada45cadf083115b27c5292fdd, data reload: false
query5 4328 654 531 531
query6 329 227 205 205
query7 4232 570 308 308
query8 319 228 222 222
query9 8830 3933 3946 3933
query10 443 341 308 308
query11 5803 2402 2217 2217
query12 184 125 139 125
query13 1256 651 402 402
query14 5919 5384 5058 5058
query14_1 4299 4316 4344 4316
query15 213 201 177 177
query16 1009 468 439 439
query17 946 716 579 579
query18 2435 496 348 348
query19 215 202 167 167
query20 135 139 127 127
query21 212 133 112 112
query22 13663 13634 13381 13381
query23 17210 16291 16121 16121
query23_1 16169 16173 16167 16167
query24 7385 1758 1294 1294
query24_1 1297 1319 1316 1316
query25 589 502 455 455
query26 1325 310 169 169
query27 2732 554 329 329
query28 4519 1937 1922 1922
query29 970 631 497 497
query30 300 233 197 197
query31 1114 1062 937 937
query32 86 75 71 71
query33 536 350 298 298
query34 1183 1087 621 621
query35 765 797 694 694
query36 1303 1355 1217 1217
query37 157 102 89 89
query38 3220 3147 3091 3091
query39 930 921 905 905
query39_1 884 885 869 869
query40 228 151 128 128
query41 77 63 61 61
query42 110 108 111 108
query43 329 325 289 289
query44
query45 209 199 194 194
query46 1072 1241 697 697
query47 2287 2312 2257 2257
query48 408 415 283 283
query49 632 484 383 383
query50 1006 335 244 244
query51 4362 4283 4339 4283
query52 109 106 94 94
query53 247 283 199 199
query54 318 271 253 253
query55 95 92 85 85
query56 315 315 324 315
query57 1415 1420 1321 1321
query58 292 271 264 264
query59 1548 1608 1398 1398
query60 331 323 345 323
query61 160 156 155 155
query62 675 624 561 561
query63 236 200 200 200
query64 2422 798 644 644
query65
query66 1734 468 350 350
query67 29404 29951 29710 29710
query68
query69 475 341 286 286
query70 1019 1006 989 989
query71 302 274 274 274
query72 3009 2734 2593 2593
query73 814 822 451 451
query74 5104 4911 4740 4740
query75 2675 2624 2292 2292
query76 2282 1128 771 771
query77 409 417 340 340
query78 12146 11952 11698 11698
query79 1287 1071 723 723
query80 614 578 487 487
query81 455 284 248 248
query82 231 159 124 124
query83 284 288 256 256
query84 297 144 116 116
query85 961 614 550 550
query86 363 343 328 328
query87 3409 3381 3214 3214
query88 3530 2666 2635 2635
query89 419 388 334 334
query90 2165 177 168 168
query91 178 167 141 141
query92 82 78 72 72
query93 1500 1323 836 836
query94 530 372 305 305
query95 673 375 376 375
query96 1055 805 325 325
query97 2670 2675 2543 2543
query98 233 231 226 226
query99 1089 1113 976 976
Total cold run time: 250590 ms
Total hot run time: 169132 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4532598729
<details>
<summary>TPC-H: <b>Total hot run time: 31320 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a8e496ea1def13252d8ee1f5793244d1e90efa49, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17726 3993 3965 3965
q2 q3 10784 1400 832 832
q4 4689 473 351 351
q5 7571 2283 2167 2167
q6 329 174 136 136
q7 958 798 648 648
q8 9363 1702 1528 1528
q9 6641 4992 4938 4938
q10 6429 2242 1921 1921
q11 443 277 245 245
q12 693 425 299 299
q13 18198 3449 2752 2752
q14 276 257 239 239
q15 q16 819 769 702 702
q17 1052 982 905 905
q18 6988 5748 5654 5654
q19 1242 1347 1159 1159
q20 525 421 257 257
q21 5710 2550 2308 2308
q22 443 362 314 314
Total cold run time: 100879 ms
Total hot run time: 31320 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4385 4271 4285 4271
q2 q3 4542 4920 4351 4351
q4 2117 2227 1399 1399
q5 4448 4372 4746 4372
q6 261 197 150 150
q7 2042 1911 1599 1599
q8 2548 2215 2211 2211
q9 8061 8017 8100 8017
q10 4830 4757 4458 4458
q11 643 430 386 386
q12 769 761 549 549
q13 3306 3726 3014 3014
q14 298 306 264 264
q15 q16 758 745 652 652
q17 1402 1375 1373 1373
q18 7980 7419 6755 6755
q19 1094 1095 1093 1093
q20 2209 2220 1963 1963
q21 5315 4641 4432 4432
q22 530 482 408 408
Total cold run time: 57538 ms
Total hot run time: 51717 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4540478372
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (3/3)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63529_97420ea7db825a94f0d23e5c17aff8af887dbfdb_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63529_97420ea7db825a94f0d23e5c17aff8af887dbfdb_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.72% (28010/37997) |
| Line Coverage | 57.68% (304286/527586) |
| Region Coverage | 54.75% (254214/464345) |
| Branch Coverage | 56.30% (109893/195208) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Preserve shuffle for serial merge aggregation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63529:
URL: https://github.com/apache/doris/pull/63529#issuecomment-4540605294
<details>
<summary>TPC-DS: <b>Total hot run time: 172747 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 97420ea7db825a94f0d23e5c17aff8af887dbfdb, data reload: false
query5 4314 670 522 522
query6 330 229 216 216
query7 4226 589 302 302
query8 331 233 222 222
query9 8848 4116 4095 4095
query10 450 338 302 302
query11 5813 2539 2258 2258
query12 181 128 130 128
query13 1296 612 435 435
query14 6224 5554 5224 5224
query14_1 4534 4544 4581 4544
query15 215 204 182 182
query16 1015 487 451 451
query17 1126 729 611 611
query18 2565 492 368 368
query19 211 200 165 165
query20 135 132 125 125
query21 212 138 115 115
query22 13595 13732 13391 13391
query23 17508 16652 16223 16223
query23_1 16395 16450 16542 16450
query24 7691 1794 1325 1325
query24_1 1375 1325 1332 1325
query25 555 476 420 420
query26 1285 324 173 173
query27 2710 556 342 342
query28 4445 2010 2013 2010
query29 999 638 499 499
query30 307 238 198 198
query31 1146 1084 947 947
query32 93 81 74 74
query33 573 362 314 314
query34 1163 1138 667 667
query35 773 804 704 704
query36 1417 1401 1288 1288
query37 162 104 89 89
query38 3235 3201 3081 3081
query39 939 929 909 909
query39_1 885 916 908 908
query40 249 149 134 134
query41 73 68 68 68
query42 113 110 119 110
query43 335 336 295 295
query44
query45 224 211 208 208
query46 1106 1216 741 741
query47 2406 2362 2334 2334
query48 420 429 341 341
query49 681 520 406 406
query50 1051 362 254 254
query51 4434 4307 4329 4307
query52 105 107 102 102
query53 259 285 208 208
query54 334 299 271 271
query55 95 93 86 86
query56 319 327 335 327
query57 1456 1424 1338 1338
query58 313 287 272 272
query59 1603 1702 1497 1497
query60 346 340 331 331
query61 190 183 187 183
query62 722 655 592 592
query63 253 239 211 211
query64 2402 792 637 637
query65
query66 1700 483 361 361
query67 29825 29726 29562 29562
query68
query69 481 349 309 309
query70 1067 1027 1007 1007
query71 311 269 265 265
query72 3126 2731 2396 2396
query73 855 828 460 460
query74 5126 4952 4833 4833
query75 2704 2625 2264 2264
query76 2294 1140 811 811
query77 412 419 348 348
query78 12531 12422 11794 11794
query79 1472 1013 749 749
query80 646 552 449 449
query81 450 281 233 233
query82 1352 160 124 124
query83 358 278 268 268
query84 269 141 112 112
query85 888 522 443 443
query86 402 321 324 321
query87 3448 3383 3234 3234
query88 3613 2743 2712 2712
query89 439 394 344 344
query90 1951 182 176 176
query91 179 172 139 139
query92 78 74 75 74
query93 1540 1435 916 916
query94 527 359 288 288
query95 664 387 444 387
query96 1044 790 339 339
query97 2777 2763 2611 2611
query98 228 229 228 228
query99 1173 1152 1044 1044
Total cold run time: 255751 ms
Total hot run time: 172747 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org