You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "wangbo (via GitHub)" <gi...@apache.org> on 2024/01/29 11:59:59 UTC
[PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
wangbo opened a new pull request, #30529:
URL: https://github.com/apache/doris/pull/30529
## Proposed changes
ThreadPoolToken not support call wait and run in one threadpool, while we want to put wait/run in one threadpool later to using workload group to limit cpu.
So we can implement ThreadPoolToken's function in FlushToken and remove ThreadPoolToken's.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #30529:
URL: https://github.com/apache/doris/pull/30529
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1914750476
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1914679049
<details>
<summary>ClickBench: <b>Total hot run time: 31 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 556cd77f07b908c6b0400cef788ddb74f30582d2, data reload: false
query1 0.04 0.03 0.03
query2 0.05 0.02 0.02
query3 0.23 0.06 0.06
query4 1.66 0.10 0.10
query5 0.53 0.52 0.51
query6 1.21 0.65 0.65
query7 0.02 0.01 0.01
query8 0.04 0.02 0.03
query9 0.56 0.50 0.49
query10 0.57 0.57 0.55
query11 0.12 0.09 0.08
query12 0.10 0.08 0.09
query13 0.60 0.61 0.60
query14 0.79 0.79 0.80
query15 0.79 0.77 0.77
query16 0.38 0.37 0.39
query17 1.05 1.02 0.99
query18 0.20 0.26 0.24
query19 1.84 1.79 1.76
query20 0.02 0.01 0.01
query21 15.42 0.60 0.59
query22 2.67 1.97 1.57
query23 17.31 0.76 0.89
query24 2.26 1.34 1.29
query25 0.36 0.14 0.14
query26 0.68 0.14 0.14
query27 0.06 0.05 0.05
query28 12.00 0.84 0.87
query29 12.64 3.26 3.09
query30 0.64 0.54 0.56
query31 2.80 0.34 0.36
query32 3.35 0.50 0.48
query33 3.21 3.26 3.19
query34 15.91 4.37 4.28
query35 4.27 4.27 4.31
query36 1.11 1.07 1.05
query37 0.06 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.01 0.02
query40 0.17 0.12 0.13
query41 0.08 0.02 0.01
query42 0.02 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.91 s
Total hot run time: 31 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1914684272
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 556cd77f07b908c6b0400cef788ddb74f30582d2 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.4 seconds inserted 10000000 Rows, about 694K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30529:
URL: https://github.com/apache/doris/pull/30529#discussion_r1469499673
##########
be/src/olap/memtable_flush_executor.h:
##########
@@ -75,15 +75,20 @@ class FlushToken {
const MemTableStat& memtable_stat() { return _memtable_stat; }
+private:
+ void _run_memtable_flush_task(std::unique_ptr<MemTable> memtable_ptr, int32_t segment_id,
+ int64_t submit_task_time);
+ void _shutdown_flush_token() { _shutdown.store(true); }
+ bool _is_shutdown() { return _shutdown.load(); }
+ void _wait_running_task_finish();
+
private:
Review Comment:
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
```suggestion
```
<details>
<summary>Additional context</summary>
**be/src/olap/memtable_flush_executor.h:77:** previously declared here
```cpp
private:
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1914584080
TeamCity be ut coverage result:
Function Coverage: 36.20% (8625/23828)
Line Coverage: 28.26% (70538/249601)
Region Coverage: 27.25% (36388/133534)
Branch Coverage: 24.06% (18653/77540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4da00b17ded7ab55fb68e4d5804bd8a5d3da9a6c_4da00b17ded7ab55fb68e4d5804bd8a5d3da9a6c/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1914610835
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1914669662
<details>
<summary>TPC-DS: <b>Total hot run time: 172557 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 556cd77f07b908c6b0400cef788ddb74f30582d2, data reload: false
query1 923 329 318 318
query2 6577 2028 1825 1825
query3 6692 199 192 192
query4 34149 22067 22020 22020
query5 4449 431 431 431
query6 238 166 152 152
query7 4601 262 260 260
query8 241 177 177 177
query9 8574 2281 2252 2252
query10 416 203 200 200
query11 17555 15310 15259 15259
query12 127 66 65 65
query13 1699 381 363 363
query14 9222 6412 6322 6322
query15 208 182 179 179
query16 5699 254 245 245
query17 957 481 468 468
query18 1814 258 250 250
query19 171 130 134 130
query20 74 67 64 64
query21 209 132 128 128
query22 5026 4885 4852 4852
query23 31271 30509 30363 30363
query24 12547 2736 2803 2736
query25 607 315 305 305
query26 1901 139 140 139
query27 3186 285 276 276
query28 7765 1837 1814 1814
query29 1983 617 596 596
query30 286 133 133 133
query31 898 713 710 710
query32 77 54 51 51
query33 701 208 211 208
query34 1183 456 466 456
query35 870 747 727 727
query36 1337 1139 1157 1139
query37 96 59 55 55
query38 3259 3176 3137 3137
query39 1307 1258 1245 1245
query40 348 82 86 82
query41 38 35 35 35
query42 88 80 82 80
query43 494 482 489 482
query44 1091 705 692 692
query45 200 174 175 174
query46 1043 664 641 641
query47 1588 1492 1511 1492
query48 402 303 311 303
query49 1182 285 283 283
query50 689 312 318 312
query51 5239 5149 5108 5108
query52 88 78 81 78
query53 318 279 260 260
query54 240 192 186 186
query55 86 76 78 76
query56 179 165 162 162
query57 962 900 901 900
query58 184 156 161 156
query59 2728 2425 2564 2425
query60 204 174 184 174
query61 84 88 86 86
query62 589 372 372 372
query63 294 270 261 261
query64 5998 1736 1721 1721
query65 3278 3232 3191 3191
query66 1261 325 313 313
query67 14610 14123 14202 14123
query68 3749 476 484 476
query69 425 300 297 297
query70 1496 1482 1510 1482
query71 281 215 209 209
query72 3752 2813 2845 2813
query73 673 321 318 318
query74 6558 6178 6216 6178
query75 2799 2268 2288 2268
query76 2656 983 968 968
query77 381 238 230 230
query78 9091 8891 8537 8537
query79 2898 490 482 482
query80 1995 318 315 315
query81 505 195 197 195
query82 735 74 79 74
query83 237 124 116 116
query84 288 75 71 71
query85 2321 348 323 323
query86 511 403 371 371
query87 3448 3281 3241 3241
query88 4162 2181 2191 2181
query89 424 374 364 364
query90 1937 189 184 184
query91 149 129 116 116
query92 50 43 41 41
query93 4457 409 431 409
query94 1375 157 160 157
query95 504 467 454 454
query96 636 321 314 314
query97 4244 4105 4119 4105
query98 214 187 187 187
query99 1047 697 671 671
Total cold run time: 286398 ms
Total hot run time: 172557 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1915934994
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "wangbo (via GitHub)" <gi...@apache.org>.
wangbo commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1914549081
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1914650784
<details>
<summary>TPC-H: <b>Total hot run time: 37541 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 556cd77f07b908c6b0400cef788ddb74f30582d2, data reload: false
------ Round 1 ----------------------------------
q1 7148 5373 4922 4922
q2 2040 137 129 129
q3 10632 920 909 909
q4 4788 766 694 694
q5 7670 2902 2820 2820
q6 186 121 121 121
q7 1132 726 708 708
q8 9357 2001 2011 2001
q9 7250 6285 6266 6266
q10 8103 2387 2449 2387
q11 418 200 193 193
q12 738 280 275 275
q13 18006 3336 3243 3243
q14 284 270 250 250
q15 530 498 491 491
q16 470 405 413 405
q17 933 564 518 518
q18 7138 6114 6118 6114
q19 3055 1389 1397 1389
q20 583 324 324 324
q21 6768 3102 3092 3092
q22 802 304 290 290
Total cold run time: 98031 ms
Total hot run time: 37541 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5093 5025 4767 4767
q2 351 239 228 228
q3 2924 2701 2731 2701
q4 1793 1682 1584 1584
q5 5288 5208 5043 5043
q6 202 117 120 117
q7 2088 1805 1762 1762
q8 3109 3242 3182 3182
q9 8684 8459 8414 8414
q10 5458 3471 3359 3359
q11 566 460 473 460
q12 738 573 595 573
q13 16945 3113 3076 3076
q14 274 253 266 253
q15 537 485 482 482
q16 528 478 477 477
q17 1976 1706 1768 1706
q18 8911 11466 8821 8821
q19 24033 1608 1539 1539
q20 4622 1898 1888 1888
q21 15368 4742 4865 4742
q22 1951 501 459 459
Total cold run time: 111439 ms
Total hot run time: 55633 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](executor)Remove ThreadPoolToken from MemTableFlushExecutor [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30529:
URL: https://github.com/apache/doris/pull/30529#issuecomment-1914657749
TeamCity be ut coverage result:
Function Coverage: 36.19% (8622/23827)
Line Coverage: 28.24% (70496/249596)
Region Coverage: 27.23% (36363/133534)
Branch Coverage: 24.04% (18637/77540)
Coverage Report: http://coverage.selectdb-in.cc/coverage/556cd77f07b908c6b0400cef788ddb74f30582d2_556cd77f07b908c6b0400cef788ddb74f30582d2/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org