You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2024/06/05 08:59:17 UTC
[PR] [fix](statistics)Fix partition table health rate evaluation bug. [doris]
Jibing-Li opened a new pull request, #35924:
URL: https://github.com/apache/doris/pull/35924
The older Doris version doesn't support partition stats collection, so when user upgrade their cluster, the metadata doesn't contain partition level update rows for each table. In this case, auto collector thought this table haven't collect partition stats, which will trigger a new collection. Since the partition level update rows are only updated after load operation, if user stop loading, auto analyzing will keep collecting this table for ever.
This pr is to fix this problem, set the partition update rows to 0 for the old version tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Fix partition table health rate evaluation bug. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35924:
URL: https://github.com/apache/doris/pull/35924#issuecomment-2149261842
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Fix partition table health rate evaluation bug. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li merged PR #35924:
URL: https://github.com/apache/doris/pull/35924
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Fix partition table health rate evaluation bug. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35924:
URL: https://github.com/apache/doris/pull/35924#issuecomment-2149559623
<details>
<summary>TPC-H: <b>Total hot run time: 40469 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 14a14a59648f83a1f2303ae0683e426345c96f51, data reload: false
------ Round 1 ----------------------------------
q1 17930 4681 4344 4344
q2 2990 194 203 194
q3 11793 1151 1153 1151
q4 10430 768 754 754
q5 7939 2662 2618 2618
q6 226 142 135 135
q7 968 599 586 586
q8 9225 2051 2099 2051
q9 8826 6485 6449 6449
q10 8823 3723 3700 3700
q11 438 240 234 234
q12 446 233 227 227
q13 17762 3015 2942 2942
q14 275 215 227 215
q15 503 463 464 463
q16 531 379 375 375
q17 968 669 684 669
q18 8033 7461 7423 7423
q19 7623 1423 1513 1423
q20 629 306 306 306
q21 5026 3877 3979 3877
q22 391 339 333 333
Total cold run time: 121775 ms
Total hot run time: 40469 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4387 4248 4198 4198
q2 376 261 271 261
q3 2966 2759 2785 2759
q4 1857 1646 1680 1646
q5 5245 5309 5276 5276
q6 216 124 125 124
q7 2088 1779 1756 1756
q8 3214 3283 3339 3283
q9 8272 8346 8341 8341
q10 3872 3695 3671 3671
q11 581 497 489 489
q12 779 586 634 586
q13 17232 2938 2988 2938
q14 303 258 286 258
q15 505 477 478 477
q16 505 410 410 410
q17 1777 1480 1493 1480
q18 7517 7461 7419 7419
q19 1638 1600 1580 1580
q20 1949 1814 1808 1808
q21 4799 4664 4807 4664
q22 627 563 523 523
Total cold run time: 70705 ms
Total hot run time: 53947 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Fix partition table health rate evaluation bug. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #35924:
URL: https://github.com/apache/doris/pull/35924#issuecomment-2149490933
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Fix partition table health rate evaluation bug. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35924:
URL: https://github.com/apache/doris/pull/35924#issuecomment-2149588309
<details>
<summary>TPC-DS: <b>Total hot run time: 170853 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 14a14a59648f83a1f2303ae0683e426345c96f51, data reload: false
query1 929 373 361 361
query2 6466 2467 2280 2280
query3 6653 205 205 205
query4 19205 17349 17392 17349
query5 4186 450 458 450
query6 258 160 169 160
query7 4588 300 289 289
query8 336 277 273 273
query9 8471 2457 2426 2426
query10 440 299 293 293
query11 10644 10122 10066 10066
query12 137 90 86 86
query13 1628 367 364 364
query14 10202 7625 7747 7625
query15 224 190 188 188
query16 7814 288 257 257
query17 1775 512 500 500
query18 1958 275 271 271
query19 202 152 158 152
query20 92 82 85 82
query21 206 133 134 133
query22 4287 4148 4022 4022
query23 33637 33197 33109 33109
query24 12022 2879 2805 2805
query25 676 351 357 351
query26 1812 154 151 151
query27 2984 312 317 312
query28 7709 2063 2068 2063
query29 1137 641 608 608
query30 281 146 149 146
query31 971 746 740 740
query32 97 51 53 51
query33 762 288 282 282
query34 958 474 476 474
query35 738 624 618 618
query36 1118 961 907 907
query37 180 72 73 72
query38 2872 2779 2747 2747
query39 856 801 810 801
query40 276 127 126 126
query41 59 50 51 50
query42 121 99 102 99
query43 557 567 540 540
query44 1243 737 752 737
query45 196 174 164 164
query46 1082 717 725 717
query47 1817 1759 1847 1759
query48 371 292 298 292
query49 1176 412 413 412
query50 789 385 389 385
query51 6778 6700 6606 6606
query52 102 95 95 95
query53 357 291 289 289
query54 986 463 438 438
query55 75 71 74 71
query56 284 266 260 260
query57 1146 1044 1044 1044
query58 255 286 253 253
query59 3535 3294 3338 3294
query60 293 301 267 267
query61 86 89 84 84
query62 628 442 440 440
query63 326 290 286 286
query64 9751 2196 1826 1826
query65 3204 3130 3151 3130
query66 1374 328 338 328
query67 15444 15040 15024 15024
query68 4576 531 526 526
query69 451 301 295 295
query70 1094 1100 1158 1100
query71 415 286 296 286
query72 7055 5860 2707 2707
query73 768 328 319 319
query74 5929 5504 5471 5471
query75 3407 2641 2685 2641
query76 2592 930 903 903
query77 438 297 313 297
query78 10359 9685 9770 9685
query79 2267 514 511 511
query80 848 467 460 460
query81 565 221 222 221
query82 762 104 103 103
query83 279 167 172 167
query84 253 85 88 85
query85 1629 267 260 260
query86 504 282 310 282
query87 3274 3132 3082 3082
query88 3972 2357 2347 2347
query89 485 393 378 378
query90 1821 194 189 189
query91 125 93 102 93
query92 60 50 52 50
query93 2138 514 506 506
query94 1274 192 192 192
query95 408 310 319 310
query96 587 269 269 269
query97 3200 3056 2987 2987
query98 237 221 212 212
query99 1159 834 837 834
Total cold run time: 274753 ms
Total hot run time: 170853 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Fix partition table health rate evaluation bug. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35924:
URL: https://github.com/apache/doris/pull/35924#issuecomment-2149391894
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Fix partition table health rate evaluation bug. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35924:
URL: https://github.com/apache/doris/pull/35924#issuecomment-2149391978
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Fix partition table health rate evaluation bug. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35924:
URL: https://github.com/apache/doris/pull/35924#issuecomment-2149599342
<details>
<summary>ClickBench: <b>Total hot run time: 30.14 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 14a14a59648f83a1f2303ae0683e426345c96f51, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.68 0.10 0.10
query5 0.51 0.50 0.50
query6 1.12 0.72 0.72
query7 0.02 0.01 0.01
query8 0.05 0.04 0.04
query9 0.54 0.46 0.48
query10 0.53 0.54 0.53
query11 0.15 0.10 0.11
query12 0.15 0.11 0.12
query13 0.59 0.58 0.60
query14 0.79 0.76 0.78
query15 0.83 0.81 0.81
query16 0.38 0.35 0.36
query17 1.01 0.99 1.00
query18 0.22 0.24 0.27
query19 1.81 1.80 1.81
query20 0.02 0.01 0.01
query21 15.47 0.74 0.68
query22 4.94 6.89 1.52
query23 18.34 1.38 1.21
query24 1.86 0.20 0.20
query25 0.13 0.09 0.09
query26 0.27 0.17 0.18
query27 0.09 0.08 0.09
query28 13.40 1.03 1.01
query29 12.69 3.31 3.28
query30 0.24 0.06 0.05
query31 2.87 0.39 0.38
query32 3.28 0.47 0.47
query33 2.92 2.90 2.90
query34 17.01 4.38 4.40
query35 4.52 4.46 4.46
query36 0.65 0.46 0.48
query37 0.17 0.15 0.15
query38 0.15 0.13 0.13
query39 0.04 0.03 0.04
query40 0.17 0.15 0.15
query41 0.10 0.05 0.04
query42 0.05 0.04 0.04
query43 0.04 0.03 0.04
Total cold run time: 110.15 s
Total hot run time: 30.14 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org