You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mymeiyi (via GitHub)" <gi...@apache.org> on 2024/06/13 11:09:17 UTC
[PR] [fix](group commit) Fix group commit in nereids [doris]
mymeiyi opened a new pull request, #36267:
URL: https://github.com/apache/doris/pull/36267
## Proposed changes
1. The group commit in nereids use `isGroupCommit` variable in `ConnectContext` to mark if it is a group commit, but not reset. If executes 2 sqls, the first is group_commit, the second is not group_commit, the `isGroupCommit` is true for the second sql and the data_sink is `GroupCommitBlockSink`
2. Move the `isGroupCommitAvailablePlan` to the end when check `canGroupCommit` because it's cpu expensive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) Fix group commit in nereids [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36267:
URL: https://github.com/apache/doris/pull/36267#issuecomment-2165455181
<details>
<summary>TPC-DS: <b>Total hot run time: 170620 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b43a6dda734a06dec140977f564bdb946ee8942c, data reload: false
query1 931 383 374 374
query2 6460 2422 2285 2285
query3 6635 211 208 208
query4 20127 17557 17320 17320
query5 3593 463 456 456
query6 245 158 177 158
query7 4584 296 299 296
query8 319 292 298 292
query9 8518 2468 2483 2468
query10 572 310 307 307
query11 10534 10015 10012 10012
query12 120 97 89 89
query13 1661 384 382 382
query14 10198 7610 6866 6866
query15 229 188 180 180
query16 7753 274 265 265
query17 1915 543 518 518
query18 1942 276 278 276
query19 202 150 153 150
query20 94 82 84 82
query21 206 130 122 122
query22 4329 4125 3962 3962
query23 33766 33408 33588 33408
query24 9016 2912 2879 2879
query25 594 384 379 379
query26 710 159 154 154
query27 2268 325 334 325
query28 5695 2181 2140 2140
query29 906 646 617 617
query30 251 151 153 151
query31 954 775 771 771
query32 102 53 57 53
query33 641 281 280 280
query34 899 505 482 482
query35 707 640 602 602
query36 1102 964 976 964
query37 150 77 77 77
query38 2931 2815 2812 2812
query39 910 851 831 831
query40 232 128 128 128
query41 58 54 58 54
query42 124 103 100 100
query43 625 576 582 576
query44 1093 718 735 718
query45 191 158 165 158
query46 1057 695 742 695
query47 1894 1759 1786 1759
query48 365 300 295 295
query49 841 394 403 394
query50 759 401 402 401
query51 6795 6636 6575 6575
query52 103 96 93 93
query53 357 286 286 286
query54 802 462 440 440
query55 74 75 75 75
query56 296 253 255 253
query57 1109 1029 1075 1029
query58 249 249 247 247
query59 3415 3072 3224 3072
query60 287 263 327 263
query61 92 87 90 87
query62 598 443 430 430
query63 324 285 290 285
query64 8474 2258 1727 1727
query65 3154 3085 3063 3063
query66 731 328 349 328
query67 15353 15070 14855 14855
query68 4579 553 553 553
query69 621 460 380 380
query70 1142 1091 1069 1069
query71 445 323 266 266
query72 7255 5654 2739 2739
query73 760 325 326 325
query74 5852 5511 5510 5510
query75 3378 2648 2672 2648
query76 2607 949 933 933
query77 593 293 285 285
query78 10335 9931 9906 9906
query79 1381 510 507 507
query80 1358 471 465 465
query81 604 216 224 216
query82 566 104 100 100
query83 277 160 160 160
query84 255 83 88 83
query85 1076 279 267 267
query86 446 297 314 297
query87 3280 3119 3082 3082
query88 4047 2452 2425 2425
query89 470 379 373 373
query90 1677 183 192 183
query91 129 102 99 99
query92 62 50 51 50
query93 1866 504 507 504
query94 1107 193 196 193
query95 401 311 317 311
query96 589 338 278 278
query97 3276 3011 3010 3010
query98 216 202 191 191
query99 1150 833 838 833
Total cold run time: 264021 ms
Total hot run time: 170620 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) Fix group commit in nereids [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36267:
URL: https://github.com/apache/doris/pull/36267#issuecomment-2165432311
<details>
<summary>TPC-H: <b>Total hot run time: 40077 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b43a6dda734a06dec140977f564bdb946ee8942c, data reload: false
------ Round 1 ----------------------------------
q1 17643 4307 4300 4300
q2 2028 187 190 187
q3 10458 1088 1119 1088
q4 10194 675 779 675
q5 7477 2699 2641 2641
q6 230 136 139 136
q7 950 589 596 589
q8 9223 2046 2086 2046
q9 9083 6486 6466 6466
q10 9001 3713 3694 3694
q11 440 232 236 232
q12 490 231 223 223
q13 18901 2945 2974 2945
q14 267 218 239 218
q15 508 483 463 463
q16 527 381 371 371
q17 973 638 619 619
q18 7950 7334 7397 7334
q19 6518 1379 1497 1379
q20 663 315 322 315
q21 4911 3822 3973 3822
q22 395 334 343 334
Total cold run time: 118830 ms
Total hot run time: 40077 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4384 4290 4234 4234
q2 371 258 258 258
q3 3060 2904 2895 2895
q4 1969 1765 1815 1765
q5 5419 5490 5400 5400
q6 223 129 132 129
q7 2241 1866 1847 1847
q8 3292 3427 3415 3415
q9 8699 8738 8713 8713
q10 4180 3899 3837 3837
q11 619 477 479 477
q12 847 630 625 625
q13 16397 3143 3161 3143
q14 311 269 270 269
q15 530 492 492 492
q16 486 449 430 430
q17 1813 1526 1479 1479
q18 8132 7844 7695 7695
q19 1866 1720 1659 1659
q20 3137 1882 1879 1879
q21 5044 4894 4794 4794
q22 670 545 575 545
Total cold run time: 73690 ms
Total hot run time: 55980 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) Fix group commit in nereids [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36267:
URL: https://github.com/apache/doris/pull/36267#issuecomment-2165773352
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) Fix group commit in nereids [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #36267:
URL: https://github.com/apache/doris/pull/36267#issuecomment-2165346945
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) Fix group commit in nereids [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36267:
URL: https://github.com/apache/doris/pull/36267#issuecomment-2165773495
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) Fix group commit in nereids [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #36267:
URL: https://github.com/apache/doris/pull/36267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) Fix group commit in nereids [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36267:
URL: https://github.com/apache/doris/pull/36267#issuecomment-2165467077
<details>
<summary>ClickBench: <b>Total hot run time: 31.21 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b43a6dda734a06dec140977f564bdb946ee8942c, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.66 0.08 0.08
query5 0.50 0.48 0.51
query6 1.12 0.73 0.74
query7 0.03 0.02 0.02
query8 0.05 0.04 0.04
query9 0.55 0.48 0.49
query10 0.55 0.56 0.54
query11 0.15 0.11 0.10
query12 0.14 0.12 0.12
query13 0.59 0.60 0.59
query14 0.79 0.80 0.80
query15 0.83 0.81 0.81
query16 0.35 0.36 0.36
query17 1.03 1.03 0.95
query18 0.21 0.26 0.23
query19 1.79 1.73 1.70
query20 0.02 0.01 0.01
query21 15.41 0.67 0.65
query22 3.96 6.69 2.62
query23 18.25 1.34 1.37
query24 2.13 0.21 0.22
query25 0.15 0.09 0.08
query26 0.27 0.18 0.17
query27 0.08 0.08 0.07
query28 13.24 1.03 0.99
query29 12.58 3.24 3.21
query30 0.26 0.07 0.06
query31 2.84 0.38 0.38
query32 3.27 0.46 0.47
query33 2.92 2.87 2.94
query34 17.35 4.45 4.45
query35 4.49 4.47 4.46
query36 0.64 0.45 0.46
query37 0.19 0.17 0.14
query38 0.15 0.14 0.14
query39 0.04 0.03 0.04
query40 0.18 0.15 0.14
query41 0.09 0.05 0.05
query42 0.06 0.04 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.3 s
Total hot run time: 31.21 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) Fix group commit in nereids [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36267:
URL: https://github.com/apache/doris/pull/36267#issuecomment-2165340825
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org