You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2024/01/12 06:56:39 UTC
[PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Mryange opened a new pull request, #29914:
URL: https://github.com/apache/doris/pull/29914
## Proposed changes
Reason: There might be multiple BEs performing 'markOneFragmentDone,' whereas the original consideration only accounted for a single BE scenario.
Additionally, for pipelineX, the profiles sent are on a per-Fragment basis, resulting in larger profile sizes for each individual transmission.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29914:
URL: https://github.com/apache/doris/pull/29914#issuecomment-1888564032
<details>
<summary>TPC-DS: <b>Total hot run time: 178059 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0329a58b6853364746c92515260cac238f748fd, data reload: false
query1 939 341 327 327
query2 6713 1896 1971 1896
query3 6715 205 204 204
query4 26132 22211 22234 22211
query5 8567 548 602 548
query6 274 186 193 186
query7 4617 277 264 264
query8 232 202 198 198
query9 9228 2742 2830 2742
query10 479 254 244 244
query11 16270 15356 15318 15318
query12 138 75 76 75
query13 1717 386 392 386
query14 11945 7257 7287 7257
query15 242 189 199 189
query16 5385 262 249 249
query17 1900 484 482 482
query18 1961 277 257 257
query19 291 151 147 147
query20 78 82 75 75
query21 191 126 126 126
query22 5074 4587 4756 4587
query23 31830 30783 30823 30783
query24 11007 2800 2800 2800
query25 549 329 329 329
query26 1434 161 150 150
query27 2782 273 281 273
query28 6883 1846 1855 1846
query29 1079 709 661 661
query30 273 136 139 136
query31 962 768 775 768
query32 97 65 58 58
query33 700 243 232 232
query34 1010 462 483 462
query35 861 741 717 717
query36 1352 1262 1230 1230
query37 103 71 71 71
query38 3379 3294 3231 3231
query39 1342 1286 1288 1286
query40 221 95 95 95
query41 38 35 35 35
query42 109 100 98 98
query43 546 486 524 486
query44 1097 680 697 680
query45 213 187 181 181
query46 1055 663 638 638
query47 1647 1516 1568 1516
query48 401 326 316 316
query49 1143 313 314 313
query50 707 323 325 323
query51 5347 5188 5182 5182
query52 110 94 97 94
query53 357 281 286 281
query54 929 452 466 452
query55 99 89 90 89
query56 207 189 203 189
query57 972 938 941 938
query58 218 202 201 201
query59 2789 2646 2652 2646
query60 248 228 214 214
query61 86 81 81 81
query62 651 405 372 372
query63 304 283 307 283
query64 4776 1684 1661 1661
query65 3366 3256 3248 3248
query66 1071 338 326 326
query67 15754 15383 15085 15085
query68 10992 514 499 499
query69 714 381 396 381
query70 1692 1571 1533 1533
query71 573 235 259 235
query72 4999 2812 2810 2810
query73 2091 315 304 304
query74 6956 6459 6351 6351
query75 4829 2350 2296 2296
query76 6345 1096 1054 1054
query77 693 292 275 275
query78 9675 8489 8526 8489
query79 1022 514 515 514
query80 553 347 342 342
query81 466 215 207 207
query82 195 86 92 86
query83 151 128 120 120
query84 282 76 77 76
query85 1032 344 334 334
query86 384 382 363 363
query87 3592 3427 3394 3394
query88 2828 2206 2190 2190
query89 487 392 397 392
query90 1896 206 199 199
query91 164 125 137 125
query92 67 56 56 56
query93 970 427 404 404
query94 1053 190 190 190
query95 531 489 475 475
query96 612 314 307 307
query97 4309 4150 4150 4150
query98 213 193 184 184
query99 1077 701 700 700
Total cold run time: 290635 ms
Total hot run time: 178059 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29914:
URL: https://github.com/apache/doris/pull/29914#issuecomment-1888771836
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29914:
URL: https://github.com/apache/doris/pull/29914#issuecomment-1888567322
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.7 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 21.8 seconds inserted 10000000 Rows, about 458K ops/s
storage size: 17183779072 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #29914:
URL: https://github.com/apache/doris/pull/29914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29914:
URL: https://github.com/apache/doris/pull/29914#issuecomment-1888551448
<details>
<summary>TPC-H: <b>Total hot run time: 38542 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0329a58b6853364746c92515260cac238f748fd, data reload: false
------ Round 1 ----------------------------------
q1 17948 5007 4929 4929
q2 2027 140 126 126
q3 10709 1157 1176 1157
q4 10535 779 838 779
q5 7794 3077 3150 3077
q6 207 122 123 122
q7 868 497 494 494
q8 9272 1954 1983 1954
q9 7318 6466 6431 6431
q10 8283 3184 3153 3153
q11 415 210 235 210
q12 361 202 200 200
q13 18117 3407 3398 3398
q14 250 212 211 211
q15 552 526 501 501
q16 424 387 378 378
q17 936 498 553 498
q18 7277 6701 6700 6700
q19 1588 1286 1467 1286
q20 522 303 300 300
q21 2806 2345 2415 2345
q22 347 306 293 293
Total cold run time: 108556 ms
Total hot run time: 38542 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4980 4927 4932 4927
q2 324 195 199 195
q3 3314 3305 3298 3298
q4 2233 2233 2227 2227
q5 5793 5779 5772 5772
q6 192 118 121 118
q7 2285 1838 1864 1838
q8 3434 3527 3524 3524
q9 8827 8789 8756 8756
q10 3746 3808 3789 3789
q11 535 421 420 420
q12 787 595 601 595
q13 6496 3206 3208 3206
q14 293 258 260 258
q15 560 514 509 509
q16 505 475 453 453
q17 1992 1995 2002 1995
q18 8874 8313 8307 8307
q19 1623 1624 1608 1608
q20 2167 1924 1917 1917
q21 6038 5717 5630 5630
q22 556 458 463 458
Total cold run time: 65554 ms
Total hot run time: 59800 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #29914:
URL: https://github.com/apache/doris/pull/29914#issuecomment-1888529812
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29914:
URL: https://github.com/apache/doris/pull/29914#issuecomment-1888771767
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29914:
URL: https://github.com/apache/doris/pull/29914#issuecomment-1888569646
<details>
<summary>ClickBench: <b>Total hot run time: 31.07 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0329a58b6853364746c92515260cac238f748fd, data reload: false
query1 0.06 0.05 0.05
query2 0.06 0.02 0.02
query3 0.25 0.11 0.11
query4 1.76 0.13 0.12
query5 0.52 0.50 0.52
query6 1.39 0.64 0.65
query7 0.02 0.01 0.02
query8 0.04 0.02 0.02
query9 0.55 0.50 0.50
query10 0.55 0.56 0.55
query11 0.13 0.10 0.09
query12 0.12 0.10 0.09
query13 0.61 0.60 0.60
query14 0.79 0.78 0.79
query15 0.81 0.80 0.80
query16 0.34 0.36 0.36
query17 0.98 0.94 0.94
query18 0.25 0.25 0.25
query19 1.86 1.74 1.75
query20 0.01 0.01 0.01
query21 15.44 0.57 0.56
query22 2.32 2.65 2.42
query23 17.17 0.79 0.87
query24 16.15 0.56 0.59
query25 2.17 0.15 0.14
query26 0.14 0.14 0.13
query27 0.13 0.14 0.14
query28 6.99 0.86 0.81
query29 12.54 3.23 3.23
query30 0.53 0.48 0.47
query31 2.78 0.37 0.35
query32 3.36 0.49 0.49
query33 3.24 3.25 3.21
query34 16.03 4.18 4.15
query35 4.18 4.16 4.17
query36 1.09 1.03 1.03
query37 0.07 0.05 0.05
query38 0.04 0.02 0.02
query39 0.02 0.01 0.02
query40 0.17 0.13 0.14
query41 0.08 0.01 0.01
query42 0.03 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 115.8 s
Total hot run time: 31.07 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](pipelineX) fix multi be may be missing profiles [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29914:
URL: https://github.com/apache/doris/pull/29914#issuecomment-1888572590
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a0329a58b6853364746c92515260cac238f748fd with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 12.5 seconds inserted 10000000 Rows, about 800K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org