You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "kaijchen (via GitHub)" <gi...@apache.org> on 2024/01/19 09:25:48 UTC
[PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
kaijchen opened a new pull request, #30146:
URL: https://github.com/apache/doris/pull/30146
## Proposed changes
1 memtable flushing and 1 memtable waiting in the queue should be enough to keep the flush thread busy.
This PR reduces memory pressure when there are many tablets being written at same time.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30146:
URL: https://github.com/apache/doris/pull/30146#issuecomment-1902080980
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30146:
URL: https://github.com/apache/doris/pull/30146#issuecomment-1902080989
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30146:
URL: https://github.com/apache/doris/pull/30146#issuecomment-1900079748
<details>
<summary>TPC-H: <b>Total hot run time: 39004 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7ba9c7500b729b9c2127b077af33131d96c07685, data reload: false
------ Round 1 ----------------------------------
q1 17607 5233 5308 5233
q2 2044 139 131 131
q3 10653 1165 1192 1165
q4 10228 809 845 809
q5 7734 3149 3157 3149
q6 199 123 125 123
q7 864 497 486 486
q8 9212 1965 1970 1965
q9 7247 6393 6346 6346
q10 8227 3050 3071 3050
q11 416 209 226 209
q12 353 194 189 189
q13 18020 3365 3331 3331
q14 234 216 212 212
q15 552 514 500 500
q16 431 379 374 374
q17 945 500 482 482
q18 7565 6826 6820 6820
q19 1580 1413 1404 1404
q20 587 319 311 311
q21 2773 2402 2411 2402
q22 377 317 313 313
Total cold run time: 107848 ms
Total hot run time: 39004 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5413 5244 5329 5244
q2 322 212 216 212
q3 3336 3246 3195 3195
q4 2046 2053 2025 2025
q5 6109 5921 5892 5892
q6 201 120 122 120
q7 2252 1792 1890 1792
q8 3207 3372 3400 3372
q9 8962 8807 8751 8751
q10 3847 3825 3787 3787
q11 574 464 445 445
q12 815 639 617 617
q13 16908 3166 3141 3141
q14 277 258 264 258
q15 563 506 505 505
q16 512 447 466 447
q17 1859 1850 1827 1827
q18 9389 9919 9217 9217
q19 19386 1557 1542 1542
q20 4557 1935 1908 1908
q21 14206 5468 5374 5374
q22 981 540 518 518
Total cold run time: 105722 ms
Total hot run time: 60189 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30146:
URL: https://github.com/apache/doris/pull/30146#issuecomment-1900112993
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 7ba9c7500b729b9c2127b077af33131d96c07685 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.2 seconds inserted 10000000 Rows, about 704K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30146:
URL: https://github.com/apache/doris/pull/30146#issuecomment-1900080272
TeamCity be ut coverage result:
Function Coverage: 36.73% (8655/23567)
Line Coverage: 28.76% (70688/245748)
Region Coverage: 27.64% (36455/131870)
Branch Coverage: 24.36% (18654/76568)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7ba9c7500b729b9c2127b077af33131d96c07685_7ba9c7500b729b9c2127b077af33131d96c07685/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "kaijchen (via GitHub)" <gi...@apache.org>.
kaijchen commented on PR #30146:
URL: https://github.com/apache/doris/pull/30146#issuecomment-1900050442
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30146:
URL: https://github.com/apache/doris/pull/30146#issuecomment-1900098268
<details>
<summary>TPC-DS: <b>Total hot run time: 176905 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7ba9c7500b729b9c2127b077af33131d96c07685, data reload: false
query1 930 326 322 322
query2 6567 1942 1836 1836
query3 6704 198 203 198
query4 31905 22271 22326 22271
query5 6860 549 598 549
query6 247 159 156 156
query7 4600 268 259 259
query8 221 178 173 173
query9 8392 2546 2537 2537
query10 427 243 228 228
query11 16690 15504 15524 15504
query12 122 73 70 70
query13 1709 382 387 382
query14 12011 7128 6977 6977
query15 218 181 196 181
query16 5225 234 229 229
query17 1034 484 465 465
query18 1788 250 252 250
query19 175 135 130 130
query20 81 74 69 69
query21 188 131 126 126
query22 5031 4768 4781 4768
query23 31650 30856 30780 30780
query24 11467 2802 2841 2802
query25 566 338 312 312
query26 1668 145 145 145
query27 3202 274 277 274
query28 7204 1810 1799 1799
query29 1728 630 639 630
query30 285 135 137 135
query31 940 727 760 727
query32 86 52 49 49
query33 692 215 213 213
query34 1132 455 460 455
query35 880 798 760 760
query36 1393 1254 1205 1205
query37 96 59 62 59
query38 3374 3260 3173 3173
query39 1312 1274 1283 1274
query40 340 88 84 84
query41 37 35 34 34
query42 92 83 86 83
query43 500 486 448 448
query44 1119 686 687 686
query45 192 180 174 174
query46 1080 647 662 647
query47 1644 1562 1517 1517
query48 391 303 309 303
query49 1140 290 278 278
query50 691 316 317 316
query51 5346 5149 5164 5149
query52 93 78 85 78
query53 324 263 262 262
query54 462 282 286 282
query55 80 76 78 76
query56 186 178 173 173
query57 976 880 903 880
query58 199 159 168 159
query59 2844 2742 2684 2684
query60 214 188 189 188
query61 83 84 84 84
query62 634 365 360 360
query63 297 258 260 258
query64 5493 1821 1757 1757
query65 3324 3258 3240 3240
query66 1268 320 309 309
query67 15468 15652 15223 15223
query68 12343 518 495 495
query69 601 307 307 307
query70 1652 1522 1491 1491
query71 475 214 213 213
query72 4885 2839 2842 2839
query73 2727 311 311 311
query74 7774 6348 6398 6348
query75 4750 2288 2291 2288
query76 6207 1008 971 971
query77 666 225 234 225
query78 9013 8938 8741 8741
query79 1072 507 488 488
query80 586 335 322 322
query81 453 206 200 200
query82 201 86 77 77
query83 138 129 118 118
query84 279 70 71 70
query85 1079 336 320 320
query86 395 379 376 376
query87 3501 3313 3369 3313
query88 2936 2187 2176 2176
query89 443 341 350 341
query90 1956 183 184 183
query91 161 132 142 132
query92 51 44 44 44
query93 1839 442 411 411
query94 1257 163 156 156
query95 492 452 443 443
query96 612 314 321 314
query97 4248 4127 4151 4127
query98 207 183 185 183
query99 1066 713 678 678
Total cold run time: 297382 ms
Total hot run time: 176905 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30146:
URL: https://github.com/apache/doris/pull/30146#issuecomment-1900062549
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30146:
URL: https://github.com/apache/doris/pull/30146#issuecomment-1900106645
<details>
<summary>ClickBench: <b>Total hot run time: 31.43 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7ba9c7500b729b9c2127b077af33131d96c07685, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.03 0.02
query3 0.23 0.06 0.05
query4 1.75 0.07 0.07
query5 0.54 0.53 0.52
query6 1.29 0.64 0.64
query7 0.03 0.01 0.02
query8 0.04 0.03 0.02
query9 0.56 0.49 0.50
query10 0.55 0.56 0.56
query11 0.12 0.08 0.08
query12 0.12 0.10 0.09
query13 0.60 0.61 0.60
query14 0.78 0.80 0.79
query15 0.80 0.78 0.78
query16 0.40 0.36 0.40
query17 0.99 0.99 0.99
query18 0.23 0.24 0.24
query19 1.85 1.73 1.81
query20 0.01 0.01 0.02
query21 15.42 0.57 0.59
query22 2.63 2.22 2.07
query23 17.45 0.79 0.81
query24 2.41 1.32 1.61
query25 0.29 0.24 0.14
query26 0.65 0.14 0.15
query27 0.05 0.06 0.05
query28 10.81 0.76 0.76
query29 12.55 3.23 3.18
query30 0.56 0.49 0.50
query31 2.77 0.34 0.35
query32 3.40 0.50 0.49
query33 3.16 3.20 3.22
query34 15.85 4.27 4.20
query35 4.39 4.28 4.29
query36 1.13 1.07 1.07
query37 0.07 0.05 0.05
query38 0.04 0.02 0.03
query39 0.02 0.02 0.02
query40 0.16 0.13 0.13
query41 0.08 0.01 0.02
query42 0.02 0.02 0.02
query43 0.03 0.02 0.02
Total cold run time: 104.92 s
Total hot run time: 31.43 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [config](load) set default memtable_flush_running_count_limit to 2 [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #30146:
URL: https://github.com/apache/doris/pull/30146
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org