You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2024/01/14 14:59:48 UTC
[PR] [fix](multi-table-load) fix multi table load can not finish [doris]
HHoflittlefish777 opened a new pull request, #29957:
URL: https://github.com/apache/doris/pull/29957
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890978037
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893474743
TeamCity be ut coverage result:
Function Coverage: 36.67% (8648/23581)
Line Coverage: 28.73% (70619/245773)
Region Coverage: 27.62% (36436/131925)
Branch Coverage: 24.34% (18640/76590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e8b53fbf394389ba99924c201800c50b6abfec85_e8b53fbf394389ba99924c201800c50b6abfec85/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893793160
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #29957:
URL: https://github.com/apache/doris/pull/29957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890987371
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893436689
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890977359
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890990424
<details>
<summary>TPC-H: <b>Total hot run time: 40100 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false
------ Round 1 ----------------------------------
q1 17947 6301 6020 6020
q2 2052 152 151 151
q3 10865 1112 1174 1112
q4 10522 809 762 762
q5 7816 3172 3164 3164
q6 206 124 124 124
q7 872 515 477 477
q8 9262 1989 2002 1989
q9 7299 6453 6418 6418
q10 8218 3043 3078 3043
q11 418 205 212 205
q12 363 193 191 191
q13 18100 3389 3435 3389
q14 250 221 210 210
q15 550 524 533 524
q16 432 393 384 384
q17 948 518 542 518
q18 7563 6988 7022 6988
q19 1618 1378 1402 1378
q20 564 307 280 280
q21 2892 2476 2460 2460
q22 395 330 313 313
Total cold run time: 109152 ms
Total hot run time: 40100 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5727 5582 5814 5582
q2 348 220 219 219
q3 3372 3161 3067 3067
q4 2222 2136 2031 2031
q5 6037 5933 5984 5933
q6 204 122 124 122
q7 2356 1942 2013 1942
q8 3657 3694 3790 3694
q9 9685 26064 9551 9551
q10 4145 3780 3841 3780
q11 595 449 486 449
q12 1168 650 640 640
q13 17107 3294 3247 3247
q14 310 267 275 267
q15 565 508 519 508
q16 544 475 472 472
q17 2515 2040 1994 1994
q18 9641 9475 12300 9475
q19 12792 1624 1634 1624
q20 3831 1960 1938 1938
q21 14733 5845 5602 5602
q22 980 635 594 594
Total cold run time: 102534 ms
Total hot run time: 62731 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891974119
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893744010
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893859515
<details>
<summary>TPC-H: <b>Total hot run time: 38951 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380, data reload: false
------ Round 1 ----------------------------------
q1 17716 5530 5300 5300
q2 2036 142 137 137
q3 10733 1169 1159 1159
q4 10286 789 727 727
q5 7766 3191 3196 3191
q6 197 123 120 120
q7 846 487 499 487
q8 9223 1971 1980 1971
q9 7337 6413 6389 6389
q10 8235 3043 3079 3043
q11 433 212 193 193
q12 356 190 187 187
q13 18015 3378 3329 3329
q14 242 208 216 208
q15 539 503 510 503
q16 467 373 370 370
q17 955 515 476 476
q18 7405 6911 6826 6826
q19 1827 1354 1408 1354
q20 556 309 283 283
q21 2832 2400 2446 2400
q22 348 303 298 298
Total cold run time: 108350 ms
Total hot run time: 38951 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5443 5339 5203 5203
q2 327 213 218 213
q3 3347 3216 3193 3193
q4 2089 2057 2031 2031
q5 6017 5929 5798 5798
q6 197 116 113 113
q7 2315 1849 1875 1849
q8 3222 3360 3398 3360
q9 8947 8882 8837 8837
q10 3937 3789 3775 3775
q11 554 457 441 441
q12 794 652 634 634
q13 16900 3149 3136 3136
q14 298 256 260 256
q15 548 504 505 504
q16 548 480 480 480
q17 1855 1889 1875 1875
q18 9558 19567 9377 9377
q19 24733 1557 1510 1510
q20 4605 1933 1932 1932
q21 14402 5498 5391 5391
q22 973 547 544 544
Total cold run time: 111609 ms
Total hot run time: 60452 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890993277
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.66 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.1 seconds inserted 10000000 Rows, about 473K ops/s
storage size: 17183923051 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #29957:
URL: https://github.com/apache/doris/pull/29957#discussion_r1453089406
##########
be/src/io/fs/multi_table_pipe.h:
##########
@@ -46,7 +46,13 @@ class MultiTablePipe : public KafkaConsumerPipe {
// request and execute plans for unplanned pipes
Status request_and_exec_plans();
- void set_consume_finished() { _consume_finished.store(true, std::memory_order_release); }
+ void set_consume_finished() {
+ --_inflight_cnt;
Review Comment:
why need do `--_inflight_cnt` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893492763
<details>
<summary>TPC-DS: <b>Total hot run time: 177369 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e8b53fbf394389ba99924c201800c50b6abfec85, data reload: false
query1 938 329 336 329
query2 6544 2074 1928 1928
query3 6711 199 197 197
query4 33718 22282 22172 22172
query5 6928 601 600 600
query6 264 177 174 174
query7 4607 265 263 263
query8 226 176 186 176
query9 8431 2507 2476 2476
query10 421 234 226 226
query11 16733 15341 15268 15268
query12 120 66 65 65
query13 1689 365 383 365
query14 10499 6828 6870 6828
query15 211 182 181 181
query16 4602 241 238 238
query17 980 484 477 477
query18 1789 264 259 259
query19 174 143 133 133
query20 71 70 66 66
query21 192 137 131 131
query22 5100 4859 4852 4852
query23 31517 30764 30677 30677
query24 11443 2783 2743 2743
query25 564 317 310 310
query26 1620 144 143 143
query27 3214 278 267 267
query28 7171 1851 1847 1847
query29 1395 653 684 653
query30 278 142 147 142
query31 926 746 763 746
query32 72 51 48 48
query33 691 225 215 215
query34 1115 464 462 462
query35 900 802 764 764
query36 1422 1262 1281 1262
query37 91 58 57 57
query38 3355 3224 3208 3208
query39 1310 1288 1255 1255
query40 206 92 85 85
query41 38 34 34 34
query42 89 82 89 82
query43 540 516 474 474
query44 1041 690 715 690
query45 200 181 176 176
query46 1071 657 655 655
query47 1703 1504 1553 1504
query48 400 315 312 312
query49 1132 294 286 286
query50 700 310 312 310
query51 5302 5205 5215 5205
query52 87 87 78 78
query53 338 276 281 276
query54 891 440 441 440
query55 86 88 86 86
query56 178 177 181 177
query57 999 916 938 916
query58 197 163 174 163
query59 2866 2734 2729 2729
query60 227 195 195 195
query61 84 83 89 83
query62 679 396 401 396
query63 307 265 264 264
query64 5015 1754 1735 1735
query65 3353 3251 3262 3251
query66 1244 331 320 320
query67 15927 15415 15310 15310
query68 10669 526 539 526
query69 623 303 300 300
query70 1782 1610 1603 1603
query71 548 213 225 213
query72 4834 2826 2834 2826
query73 2117 316 315 315
query74 7538 6454 6383 6383
query75 4728 2334 2277 2277
query76 6257 938 1001 938
query77 664 245 246 245
query78 9144 9040 8708 8708
query79 1031 499 495 495
query80 684 318 318 318
query81 466 204 212 204
query82 184 81 79 79
query83 146 119 118 118
query84 277 69 71 69
query85 1087 346 336 336
query86 384 397 381 381
query87 3468 3377 3345 3345
query88 3277 2190 2185 2185
query89 444 363 369 363
query90 1998 191 186 186
query91 149 124 129 124
query92 56 55 46 46
query93 3178 434 419 419
query94 1209 167 162 162
query95 498 457 454 454
query96 648 311 307 307
query97 4240 4148 4138 4138
query98 205 186 187 186
query99 989 685 685 685
Total cold run time: 296454 ms
Total hot run time: 177369 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893473982
<details>
<summary>TPC-H: <b>Total hot run time: 39614 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e8b53fbf394389ba99924c201800c50b6abfec85, data reload: false
------ Round 1 ----------------------------------
q1 18544 5709 5864 5709
q2 2419 152 141 141
q3 11946 1131 1200 1131
q4 10844 840 834 834
q5 7972 3096 3190 3096
q6 220 127 131 127
q7 915 504 485 485
q8 9215 1942 2010 1942
q9 7310 6398 6364 6364
q10 8207 3026 3057 3026
q11 403 219 205 205
q12 363 196 198 196
q13 17998 3425 3386 3386
q14 252 218 214 214
q15 555 533 503 503
q16 445 409 387 387
q17 956 485 481 481
q18 7663 6984 6966 6966
q19 1593 1307 1438 1307
q20 541 320 345 320
q21 2926 2484 2532 2484
q22 386 311 310 310
Total cold run time: 111673 ms
Total hot run time: 39614 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5868 5804 5657 5657
q2 336 223 221 221
q3 3382 3320 3255 3255
q4 2165 2099 1995 1995
q5 6165 5957 6064 5957
q6 208 123 124 123
q7 2318 1977 1923 1923
q8 3320 3505 3511 3505
q9 9390 9011 8757 8757
q10 6371 3706 3918 3706
q11 568 457 446 446
q12 863 641 665 641
q13 16958 3238 3199 3199
q14 291 263 280 263
q15 550 511 511 511
q16 544 466 475 466
q17 2031 1979 1973 1973
q18 9641 9504 15925 9504
q19 23379 1699 1634 1634
q20 4602 1963 1939 1939
q21 13809 5520 5494 5494
q22 991 570 532 532
Total cold run time: 113750 ms
Total hot run time: 61701 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890985269
<details>
<summary>TPC-DS: <b>Total hot run time: 178647 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45968bc955ad31e89f82063871a39c14c0b40e88, data reload: false
query1 939 337 328 328
query2 6736 1831 1868 1831
query3 6712 221 211 211
query4 34784 22172 22150 22150
query5 6953 527 585 527
query6 262 183 186 183
query7 4608 284 277 277
query8 233 197 201 197
query9 9164 2702 2781 2702
query10 481 243 238 238
query11 16556 15278 15193 15193
query12 130 71 70 70
query13 1690 384 381 381
query14 11259 7442 7415 7415
query15 216 184 186 184
query16 4618 260 255 255
query17 994 495 473 473
query18 1803 267 268 267
query19 180 149 147 147
query20 80 80 74 74
query21 189 122 124 122
query22 5106 4994 4799 4799
query23 31822 31845 31158 31158
query24 12709 2823 2780 2780
query25 569 328 328 328
query26 1854 152 155 152
query27 3623 288 275 275
query28 7485 1845 1840 1840
query29 2230 696 676 676
query30 283 138 145 138
query31 942 756 772 756
query32 75 63 61 61
query33 712 243 255 243
query34 1295 459 456 456
query35 899 771 757 757
query36 1394 1256 1260 1256
query37 186 67 71 67
query38 3431 3249 3244 3244
query39 1327 1273 1290 1273
query40 341 99 96 96
query41 39 35 35 35
query42 99 94 102 94
query43 558 508 531 508
query44 1109 689 697 689
query45 204 186 187 186
query46 1045 655 655 655
query47 1690 1587 1578 1578
query48 421 332 310 310
query49 1246 315 302 302
query50 703 318 309 309
query51 5312 5308 5215 5215
query52 106 86 99 86
query53 354 298 313 298
query54 897 449 451 449
query55 96 93 88 88
query56 209 196 206 196
query57 1022 969 934 934
query58 216 195 193 193
query59 2854 2574 2595 2574
query60 237 215 230 215
query61 86 95 89 89
query62 697 396 408 396
query63 299 291 296 291
query64 6250 1667 1652 1652
query65 3339 3274 3276 3274
query66 1422 335 332 332
query67 15413 15040 14931 14931
query68 12319 518 501 501
query69 657 335 338 335
query70 1702 1616 1536 1536
query71 584 259 233 233
query72 5021 2858 2902 2858
query73 3090 322 313 313
query74 7549 6407 6473 6407
query75 5680 2355 2310 2310
query76 6586 881 1094 881
query77 984 264 307 264
query78 9226 8709 8596 8596
query79 1257 503 509 503
query80 645 340 351 340
query81 464 210 208 208
query82 199 95 88 88
query83 209 122 121 121
query84 271 72 73 72
query85 1108 344 336 336
query86 383 397 384 384
query87 3605 3387 3372 3372
query88 3466 2291 2297 2291
query89 452 405 395 395
query90 2250 214 196 196
query91 153 138 135 135
query92 62 55 52 52
query93 1411 407 404 404
query94 1534 180 177 177
query95 512 490 472 472
query96 636 330 331 330
query97 4263 4168 4194 4168
query98 194 188 188 188
query99 1082 722 738 722
Total cold run time: 308347 ms
Total hot run time: 178647 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890994759
<details>
<summary>ClickBench: <b>Total hot run time: 30.94 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false
query1 0.05 0.06 0.05
query2 0.06 0.03 0.02
query3 0.26 0.12 0.11
query4 1.78 0.13 0.11
query5 0.54 0.53 0.53
query6 1.32 0.65 0.64
query7 0.02 0.01 0.01
query8 0.04 0.02 0.02
query9 0.55 0.49 0.51
query10 0.56 0.58 0.57
query11 0.12 0.09 0.09
query12 0.12 0.10 0.09
query13 0.60 0.60 0.60
query14 0.78 0.78 0.79
query15 0.80 0.78 0.77
query16 0.38 0.36 0.35
query17 1.01 0.98 1.04
query18 0.24 0.25 0.24
query19 1.89 1.81 1.78
query20 0.02 0.01 0.01
query21 15.41 0.59 0.56
query22 2.52 1.75 2.12
query23 17.29 0.77 0.87
query24 2.43 1.18 0.92
query25 1.98 0.16 0.15
query26 0.14 0.13 0.14
query27 0.14 0.15 0.15
query28 11.02 0.80 0.80
query29 12.56 3.17 3.16
query30 0.49 0.48 0.48
query31 2.78 0.35 0.35
query32 3.39 0.48 0.50
query33 3.22 3.25 3.25
query34 15.85 4.34 4.21
query35 4.25 4.23 4.23
query36 1.12 1.09 1.07
query37 0.06 0.05 0.05
query38 0.03 0.04 0.03
query39 0.02 0.01 0.02
query40 0.16 0.12 0.13
query41 0.07 0.01 0.02
query42 0.02 0.01 0.02
query43 0.03 0.01 0.02
Total cold run time: 106.12 s
Total hot run time: 30.94 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891246975
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893908188
TeamCity be ut coverage result:
Function Coverage: 36.67% (8648/23586)
Line Coverage: 28.73% (70623/245834)
Region Coverage: 27.62% (36443/131943)
Branch Coverage: 24.34% (18644/76598)
Coverage Report: http://coverage.selectdb-in.cc/coverage/838d55e9adecd7024bcf2b7aec9f4ef6aeb30380_838d55e9adecd7024bcf2b7aec9f4ef6aeb30380/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893807545
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890982261
<details>
<summary>TPC-H: <b>Total hot run time: 39943 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45968bc955ad31e89f82063871a39c14c0b40e88, data reload: false
------ Round 1 ----------------------------------
q1 18139 6046 6285 6046
q2 2051 158 135 135
q3 10667 1152 1152 1152
q4 10279 786 822 786
q5 7823 3216 3176 3176
q6 203 125 125 125
q7 900 491 484 484
q8 9296 1963 2002 1963
q9 7278 6433 6431 6431
q10 8242 3048 3099 3048
q11 422 220 209 209
q12 369 191 194 191
q13 18110 3418 3423 3418
q14 245 230 219 219
q15 554 527 512 512
q16 444 412 398 398
q17 952 503 503 503
q18 7802 7233 6808 6808
q19 1637 1329 1560 1329
q20 569 319 297 297
q21 2926 2442 2397 2397
q22 365 333 316 316
Total cold run time: 109273 ms
Total hot run time: 39943 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5849 5672 6100 5672
q2 347 223 232 223
q3 3369 3122 3141 3122
q4 2216 2027 2090 2027
q5 6139 5985 5917 5917
q6 211 128 123 123
q7 2433 2023 1968 1968
q8 3579 3658 3733 3658
q9 9770 20972 10042 10042
q10 8615 3658 3865 3658
q11 593 473 462 462
q12 1191 628 682 628
q13 17080 3262 3236 3236
q14 322 266 257 257
q15 584 508 509 508
q16 550 487 485 485
q17 3689 1966 2069 1966
q18 9612 9642 10520 9642
q19 12931 1637 1623 1623
q20 3950 1953 1940 1940
q21 13194 5727 5731 5727
q22 1537 598 622 598
Total cold run time: 107761 ms
Total hot run time: 63482 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890990402
TeamCity be ut coverage result:
Function Coverage: 36.70% (8649/23569)
Line Coverage: 28.71% (70445/245338)
Region Coverage: 27.63% (36411/131786)
Branch Coverage: 24.34% (18620/76486)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4561050a32cefa5359d52e86dd1781b0a2973274_4561050a32cefa5359d52e86dd1781b0a2973274/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890985089
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.79 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.6 seconds inserted 10000000 Rows, about 462K ops/s
storage size: 17183973713 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890985491
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891275866
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.59 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.9 seconds inserted 10000000 Rows, about 456K ops/s
storage size: 17183914922 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893882077
<details>
<summary>TPC-DS: <b>Total hot run time: 176822 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380, data reload: false
query1 937 340 328 328
query2 6564 1943 1842 1842
query3 6705 208 207 207
query4 31434 22287 22097 22097
query5 6905 597 594 594
query6 264 183 175 175
query7 4606 262 257 257
query8 227 178 177 177
query9 8374 2525 2516 2516
query10 413 239 220 220
query11 16666 15512 15593 15512
query12 117 70 74 70
query13 1706 377 370 370
query14 10490 6782 6814 6782
query15 206 172 176 172
query16 4611 241 227 227
query17 981 485 475 475
query18 1788 259 250 250
query19 168 138 130 130
query20 70 69 62 62
query21 187 127 125 125
query22 4904 4896 4809 4809
query23 32006 30812 30883 30812
query24 11484 2829 2817 2817
query25 536 312 310 310
query26 884 134 134 134
query27 2890 279 266 266
query28 6361 1802 1808 1802
query29 925 616 628 616
query30 284 136 144 136
query31 945 756 763 756
query32 69 47 48 47
query33 701 219 207 207
query34 1143 452 454 452
query35 858 762 735 735
query36 1338 1287 1290 1287
query37 91 60 58 58
query38 3428 3260 3222 3222
query39 1317 1274 1252 1252
query40 192 86 85 85
query41 38 35 35 35
query42 92 83 83 83
query43 525 500 508 500
query44 1051 677 693 677
query45 193 188 179 179
query46 1065 663 663 663
query47 1693 1560 1628 1560
query48 397 303 312 303
query49 1096 292 278 278
query50 689 304 303 303
query51 5312 5206 5172 5172
query52 83 84 75 75
query53 321 252 263 252
query54 887 428 443 428
query55 79 78 80 78
query56 170 162 172 162
query57 1011 933 954 933
query58 183 159 156 156
query59 2826 2692 2753 2692
query60 216 183 180 180
query61 84 81 93 81
query62 639 412 395 395
query63 272 259 252 252
query64 4545 1801 1766 1766
query65 3358 3279 3234 3234
query66 1233 315 306 306
query67 15810 15146 14911 14911
query68 12882 519 495 495
query69 628 300 298 298
query70 2066 1550 1488 1488
query71 554 210 202 202
query72 5422 2814 2847 2814
query73 3256 319 320 319
query74 7061 6485 6422 6422
query75 5225 2354 2297 2297
query76 6191 1018 1069 1018
query77 714 243 237 237
query78 9107 8614 8553 8553
query79 1615 481 487 481
query80 536 316 317 316
query81 468 206 206 206
query82 192 82 74 74
query83 168 120 118 118
query84 271 71 67 67
query85 1063 318 315 315
query86 388 410 382 382
query87 3545 3366 3376 3366
query88 3023 2261 2256 2256
query89 438 355 360 355
query90 2102 188 187 187
query91 156 123 123 123
query92 58 45 42 42
query93 1364 401 427 401
query94 1236 159 153 153
query95 491 454 434 434
query96 613 328 324 324
query97 4277 4133 4147 4133
query98 207 185 181 181
query99 1009 728 720 720
Total cold run time: 293969 ms
Total hot run time: 176822 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891260591
TeamCity be ut coverage result:
Function Coverage: 36.68% (8648/23577)
Line Coverage: 28.70% (70454/245455)
Region Coverage: 27.63% (36421/131818)
Branch Coverage: 24.35% (18628/76514)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4561050a32cefa5359d52e86dd1781b0a2973274_4561050a32cefa5359d52e86dd1781b0a2973274/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893892469
<details>
<summary>ClickBench: <b>Total hot run time: 30.75 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.03 0.02
query3 0.22 0.06 0.05
query4 1.69 0.07 0.06
query5 0.54 0.52 0.53
query6 1.31 0.66 0.64
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.55 0.50 0.49
query10 0.56 0.56 0.57
query11 0.11 0.08 0.08
query12 0.10 0.09 0.09
query13 0.60 0.61 0.60
query14 0.76 0.80 0.79
query15 0.80 0.78 0.76
query16 0.38 0.38 0.37
query17 1.02 0.99 1.03
query18 0.25 0.24 0.25
query19 1.82 1.75 1.79
query20 0.01 0.01 0.02
query21 15.41 0.59 0.59
query22 2.81 2.38 1.84
query23 17.31 0.79 0.74
query24 2.35 1.65 0.90
query25 0.41 0.22 0.20
query26 0.63 0.14 0.13
query27 0.06 0.05 0.05
query28 11.27 0.75 0.77
query29 12.52 3.10 3.15
query30 0.62 0.55 0.50
query31 2.77 0.33 0.35
query32 3.39 0.48 0.48
query33 3.25 3.24 3.24
query34 15.76 4.27 4.26
query35 4.24 4.28 4.23
query36 1.12 1.06 1.07
query37 0.07 0.05 0.06
query38 0.04 0.03 0.03
query39 0.02 0.01 0.01
query40 0.16 0.13 0.13
query41 0.06 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 105.19 s
Total hot run time: 30.75 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #29957:
URL: https://github.com/apache/doris/pull/29957#discussion_r1453293240
##########
be/src/io/fs/multi_table_pipe.cpp:
##########
@@ -261,20 +261,9 @@ Status MultiTablePipe::exec_plans(ExecEnv* exec_env, std::vector<ExecParam> para
_status = *status;
}
- --_inflight_plan_cnt;
- if (_inflight_plan_cnt == 0 && is_consume_finished()) {
- _ctx->number_total_rows = _number_total_rows;
- _ctx->number_loaded_rows = _number_loaded_rows;
- _ctx->number_filtered_rows = _number_filtered_rows;
- _ctx->number_unselected_rows = _number_unselected_rows;
- _ctx->commit_infos = _tablet_commit_infos;
- LOG(INFO) << "all plan for multi-table load complete. number_total_rows="
- << _ctx->number_total_rows
- << " number_loaded_rows=" << _ctx->number_loaded_rows
- << " number_filtered_rows=" << _ctx->number_filtered_rows
- << " number_unselected_rows=" << _ctx->number_unselected_rows;
- _ctx->promise.set_value(
- _status); // when all done, finish the routine load task
+ --_inflight_cnt;
+ if (_inflight_cnt == 0 && is_consume_finished()) {
Review Comment:
use fetch_sub(1) to avoid multi threads competition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1894987538
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890979279
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893501488
<details>
<summary>ClickBench: <b>Total hot run time: 30.81 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e8b53fbf394389ba99924c201800c50b6abfec85, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.03 0.02
query3 0.24 0.05 0.05
query4 1.69 0.06 0.06
query5 0.53 0.52 0.52
query6 1.25 0.66 0.63
query7 0.02 0.01 0.01
query8 0.04 0.02 0.03
query9 0.53 0.48 0.51
query10 0.56 0.57 0.56
query11 0.13 0.09 0.09
query12 0.11 0.09 0.10
query13 0.60 0.60 0.59
query14 0.77 0.78 0.79
query15 0.80 0.78 0.78
query16 0.39 0.36 0.37
query17 1.01 1.00 1.01
query18 0.24 0.24 0.26
query19 1.88 1.71 1.76
query20 0.02 0.01 0.01
query21 15.40 0.56 0.56
query22 2.48 3.12 1.56
query23 17.22 1.02 1.01
query24 2.63 1.11 1.38
query25 0.35 0.28 0.15
query26 0.60 0.14 0.14
query27 0.06 0.06 0.05
query28 10.72 0.76 0.77
query29 12.52 3.14 3.23
query30 0.55 0.49 0.47
query31 2.78 0.34 0.35
query32 3.38 0.47 0.48
query33 3.23 3.24 3.21
query34 16.07 4.29 4.22
query35 4.26 4.28 4.24
query36 1.10 1.09 1.08
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.02
query40 0.18 0.13 0.13
query41 0.07 0.02 0.01
query42 0.02 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 104.67 s
Total hot run time: 30.81 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890993420
<details>
<summary>TPC-DS: <b>Total hot run time: 179479 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false
query1 960 340 328 328
query2 6720 1912 1851 1851
query3 6708 220 212 212
query4 32297 22414 22099 22099
query5 7011 542 531 531
query6 269 197 196 196
query7 4635 282 276 276
query8 239 201 201 201
query9 9494 2757 2857 2757
query10 453 242 229 229
query11 16574 15348 15268 15268
query12 128 75 70 70
query13 1689 398 394 394
query14 11239 7387 7418 7387
query15 229 178 185 178
query16 4637 251 245 245
query17 999 500 479 479
query18 1808 269 265 265
query19 179 150 152 150
query20 81 80 75 75
query21 197 142 135 135
query22 4907 4811 4738 4738
query23 32270 31178 31084 31084
query24 12720 2914 2882 2882
query25 582 331 329 329
query26 1835 153 162 153
query27 3309 279 280 279
query28 7283 1867 1848 1848
query29 2223 692 684 684
query30 284 137 146 137
query31 941 751 762 751
query32 76 58 60 58
query33 717 241 261 241
query34 1275 470 467 467
query35 901 817 767 767
query36 1420 1308 1307 1307
query37 186 74 73 73
query38 3425 3291 3285 3285
query39 1337 1274 1274 1274
query40 345 102 99 99
query41 40 38 36 36
query42 101 102 91 91
query43 544 535 509 509
query44 1099 699 703 699
query45 205 190 181 181
query46 1072 634 652 634
query47 1623 1605 1538 1538
query48 401 315 332 315
query49 1253 307 307 307
query50 705 317 316 316
query51 5307 5244 5216 5216
query52 108 92 95 92
query53 355 292 295 292
query54 891 460 470 460
query55 95 90 90 90
query56 213 213 197 197
query57 1024 923 903 903
query58 218 198 200 198
query59 2991 2569 2691 2569
query60 234 216 221 216
query61 86 96 86 86
query62 682 423 406 406
query63 308 283 295 283
query64 6284 1680 1676 1676
query65 3359 3269 3257 3257
query66 1431 352 335 335
query67 15813 15343 15275 15275
query68 9651 550 534 534
query69 659 338 344 338
query70 1708 1604 1544 1544
query71 541 252 255 252
query72 4305 2867 2847 2847
query73 2053 323 332 323
query74 7229 6485 6466 6466
query75 4518 2344 2354 2344
query76 5782 1012 1068 1012
query77 857 279 271 271
query78 9082 8924 8621 8621
query79 1058 513 496 496
query80 737 350 361 350
query81 464 207 203 203
query82 307 102 91 91
query83 175 125 133 125
query84 279 80 73 73
query85 1138 341 329 329
query86 388 382 391 382
query87 3504 3381 3386 3381
query88 3564 2313 2294 2294
query89 464 385 397 385
query90 2069 210 202 202
query91 162 122 138 122
query92 64 54 55 54
query93 1618 412 426 412
query94 1282 182 182 182
query95 515 488 473 473
query96 634 331 331 331
query97 4287 4156 4262 4156
query98 206 183 183 183
query99 1027 756 744 744
Total cold run time: 299351 ms
Total hot run time: 179479 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893449446
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893898302
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on code in PR #29957:
URL: https://github.com/apache/doris/pull/29957#discussion_r1453089406
##########
be/src/io/fs/multi_table_pipe.h:
##########
@@ -46,7 +46,13 @@ class MultiTablePipe : public KafkaConsumerPipe {
// request and execute plans for unplanned pipes
Status request_and_exec_plans();
- void set_consume_finished() { _consume_finished.store(true, std::memory_order_release); }
+ void set_consume_finished() {
+ --_inflight_cnt;
Review Comment:
why need do `--_inflight_cnt` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893757891
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1893506230
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit e8b53fbf394389ba99924c201800c50b6abfec85 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.9 seconds inserted 10000000 Rows, about 671K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1890975923
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](multi-table-load) fix multi table load can not finish [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29957:
URL: https://github.com/apache/doris/pull/29957#issuecomment-1891264947
<details>
<summary>TPC-H: <b>Total hot run time: 40008 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false
------ Round 1 ----------------------------------
q1 18053 6242 5932 5932
q2 2041 150 132 132
q3 10910 1193 1165 1165
q4 10544 809 802 802
q5 7829 3159 3178 3159
q6 204 124 127 124
q7 889 518 489 489
q8 9294 2009 1998 1998
q9 7270 6444 6388 6388
q10 8233 3090 3089 3089
q11 416 207 222 207
q12 364 192 192 192
q13 18099 3477 3423 3423
q14 257 220 218 218
q15 562 515 511 511
q16 461 395 377 377
q17 944 510 500 500
q18 7656 7012 6939 6939
q19 1612 1304 1355 1304
q20 538 307 293 293
q21 2818 2534 2450 2450
q22 389 328 316 316
Total cold run time: 109383 ms
Total hot run time: 40008 ms
----- Round 2, with runtime_filter_mode=off -----
q1 6009 5806 5675 5675
q2 355 228 222 222
q3 3284 3122 3115 3115
q4 2251 2008 2089 2008
q5 6068 5944 6022 5944
q6 207 126 121 121
q7 2455 2050 1973 1973
q8 3705 3659 3874 3659
q9 9728 25503 9606 9606
q10 10511 3821 3901 3821
q11 721 461 482 461
q12 1498 658 673 658
q13 17356 3256 3235 3235
q14 394 256 274 256
q15 568 509 514 509
q16 564 477 473 473
q17 9324 1962 1960 1960
q18 9529 10413 9511 9511
q19 14488 1573 1617 1573
q20 4709 1956 1976 1956
q21 16303 5616 5709 5616
q22 3032 618 652 618
Total cold run time: 123059 ms
Total hot run time: 62970 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org