You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2024/07/29 07:31:11 UTC
[PR] [improvement](drop tablet) reduce shard lock wait time [doris]
yujun777 opened a new pull request, #38469:
URL: https://github.com/apache/doris/pull/38469
drop tablet's code:
```
lock_guard(tablet_manager.shard.lock);
delete tablet from tablet map
{
lock_guard(tablet.meta_lock)
...
}
...
```
Sometimes, tablet meta lock may held by other threads for a long time, it will cause shard.lock also wait for a long time.
For creating tablet thread, it also need to acquire the shard.lock, so creating tablet thread will also wait for a long time, then creating partition will be timeout.
Fix: shard.lock don't wait the time of the tablet meta lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](drop tablet) reduce shard lock wait time [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #38469:
URL: https://github.com/apache/doris/pull/38469#issuecomment-2255181059
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](drop tablet) reduce shard lock wait time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38469:
URL: https://github.com/apache/doris/pull/38469#issuecomment-2255248054
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](drop tablet) reduce shard lock wait time [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #38469:
URL: https://github.com/apache/doris/pull/38469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](drop tablet) reduce shard lock wait time [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38469:
URL: https://github.com/apache/doris/pull/38469#issuecomment-2255302523
<details>
<summary>TPC-H: <b>Total hot run time: 41752 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f565da99fb6c371cc3ea36a639d3453b1a99874, data reload: false
------ Round 1 ----------------------------------
q1 17714 4117 4091 4091
q2 2015 215 214 214
q3 10450 1328 1357 1328
q4 10167 864 965 864
q5 7642 2995 2955 2955
q6 225 140 140 140
q7 1038 617 615 615
q8 9453 1807 1938 1807
q9 8509 6595 6608 6595
q10 8741 3821 3846 3821
q11 427 266 257 257
q12 409 225 224 224
q13 17754 2943 2922 2922
q14 265 240 250 240
q15 516 476 481 476
q16 520 387 385 385
q17 959 931 922 922
q18 8012 7462 7332 7332
q19 1384 1211 1216 1211
q20 558 325 347 325
q21 5260 4745 4785 4745
q22 351 283 293 283
Total cold run time: 112369 ms
Total hot run time: 41752 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4078 4046 4025 4025
q2 323 223 223 223
q3 2991 2985 3157 2985
q4 2007 2043 1956 1956
q5 5809 5496 5456 5456
q6 221 130 131 130
q7 2133 1773 1832 1773
q8 3292 3392 3350 3350
q9 8723 8670 8793 8670
q10 3941 4111 3905 3905
q11 539 482 467 467
q12 824 610 613 610
q13 16436 3172 3123 3123
q14 306 272 280 272
q15 528 488 480 480
q16 478 437 432 432
q17 1768 1767 1744 1744
q18 8146 7746 7699 7699
q19 1718 1730 1736 1730
q20 2068 1857 1853 1853
q21 5775 5603 5241 5241
q22 534 464 461 461
Total cold run time: 72638 ms
Total hot run time: 56585 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](drop tablet) reduce shard lock wait time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38469:
URL: https://github.com/apache/doris/pull/38469#issuecomment-2256036239
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](drop tablet) reduce shard lock wait time [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38469:
URL: https://github.com/apache/doris/pull/38469#issuecomment-2255210371
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](drop tablet) reduce shard lock wait time [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38469:
URL: https://github.com/apache/doris/pull/38469#issuecomment-2255323892
<details>
<summary>TPC-DS: <b>Total hot run time: 170422 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f565da99fb6c371cc3ea36a639d3453b1a99874, data reload: false
query1 934 373 373 373
query2 6469 1766 1696 1696
query3 6675 213 223 213
query4 19534 17458 17291 17291
query5 3606 523 522 522
query6 275 180 164 164
query7 4604 295 314 295
query8 272 205 201 201
query9 8506 2422 2425 2422
query10 470 280 276 276
query11 10584 9987 10065 9987
query12 123 93 89 89
query13 1651 393 373 373
query14 9634 8731 7148 7148
query15 206 167 173 167
query16 6948 468 438 438
query17 963 598 539 539
query18 1907 288 278 278
query19 194 142 142 142
query20 92 88 88 88
query21 199 103 103 103
query22 4123 3960 3964 3960
query23 33838 33632 33447 33447
query24 10189 3242 3097 3097
query25 721 402 434 402
query26 1702 157 159 157
query27 2962 293 288 288
query28 7677 2039 2021 2021
query29 1302 456 491 456
query30 228 160 162 160
query31 945 794 778 778
query32 106 54 55 54
query33 675 319 335 319
query34 926 513 509 509
query35 867 787 778 778
query36 1044 899 897 897
query37 297 86 82 82
query38 2957 2846 2832 2832
query39 929 807 797 797
query40 252 113 114 113
query41 49 45 45 45
query42 131 109 104 104
query43 472 455 431 431
query44 1162 730 728 728
query45 209 181 179 179
query46 1096 825 821 821
query47 1780 1671 1713 1671
query48 365 296 294 294
query49 924 445 420 420
query50 928 439 443 439
query51 6636 6774 6720 6720
query52 105 91 95 91
query53 260 186 187 186
query54 650 463 489 463
query55 78 77 75 75
query56 269 262 263 262
query57 1159 1065 1073 1065
query58 270 276 275 275
query59 2619 2443 2508 2443
query60 297 277 301 277
query61 104 132 97 97
query62 888 680 667 667
query63 207 182 193 182
query64 5640 1927 1928 1927
query65 3193 3123 3093 3093
query66 1316 377 329 329
query67 15232 14908 14804 14804
query68 4233 566 596 566
query69 432 319 327 319
query70 1097 1092 1041 1041
query71 413 291 283 283
query72 7051 2762 2525 2525
query73 766 329 335 329
query74 5984 5660 5550 5550
query75 3385 2765 2736 2736
query76 2446 1339 1411 1339
query77 426 316 313 313
query78 9549 8833 9010 8833
query79 1950 546 538 538
query80 1151 516 516 516
query81 563 232 229 229
query82 1043 131 129 129
query83 237 172 169 169
query84 278 80 82 80
query85 1314 382 309 309
query86 421 301 303 301
query87 3277 3110 3086 3086
query88 3077 2513 2498 2498
query89 376 299 289 289
query90 1738 200 198 198
query91 129 105 104 104
query92 62 52 53 52
query93 1615 629 636 629
query94 816 307 304 304
query95 392 283 283 283
query96 611 289 290 289
query97 3192 3032 3040 3032
query98 220 207 204 204
query99 1635 1309 1321 1309
Total cold run time: 261504 ms
Total hot run time: 170422 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](drop tablet) reduce shard lock wait time [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38469:
URL: https://github.com/apache/doris/pull/38469#issuecomment-2255179385
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](drop tablet) reduce shard lock wait time [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38469:
URL: https://github.com/apache/doris/pull/38469#issuecomment-2255334946
<details>
<summary>ClickBench: <b>Total hot run time: 30.11 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f565da99fb6c371cc3ea36a639d3453b1a99874, data reload: false
query1 0.04 0.03 0.04
query2 0.07 0.04 0.04
query3 0.22 0.05 0.05
query4 1.69 0.06 0.06
query5 0.48 0.48 0.48
query6 1.13 0.72 0.71
query7 0.02 0.02 0.02
query8 0.05 0.04 0.05
query9 0.58 0.52 0.52
query10 0.56 0.58 0.57
query11 0.15 0.12 0.12
query12 0.15 0.13 0.13
query13 0.61 0.60 0.60
query14 0.80 0.79 0.81
query15 0.91 0.85 0.85
query16 0.35 0.36 0.35
query17 1.01 1.01 1.01
query18 0.23 0.22 0.21
query19 1.81 1.83 1.70
query20 0.01 0.01 0.01
query21 15.39 0.78 0.66
query22 3.86 8.37 1.26
query23 17.98 1.38 1.31
query24 2.28 0.23 0.22
query25 0.19 0.10 0.09
query26 0.32 0.21 0.20
query27 0.45 0.24 0.24
query28 13.16 1.00 0.96
query29 12.52 3.36 3.34
query30 0.25 0.06 0.06
query31 2.87 0.43 0.41
query32 3.23 0.49 0.49
query33 2.96 2.97 2.98
query34 15.40 4.25 4.23
query35 4.31 4.33 4.30
query36 0.67 0.47 0.48
query37 0.20 0.16 0.17
query38 0.16 0.15 0.15
query39 0.04 0.04 0.04
query40 0.16 0.13 0.13
query41 0.09 0.05 0.04
query42 0.06 0.05 0.05
query43 0.06 0.04 0.04
Total cold run time: 107.48 s
Total hot run time: 30.11 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org