You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2024/01/18 06:38:05 UTC
[PR] [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [doris]
Jibing-Li opened a new pull request, #30096:
URL: https://github.com/apache/doris/pull/30096
Sample analyze may write 0 result if getRowCount is not updated while analyzing. So we need to reanalyze the table if getRowCount > 0 and previous analyze row count is 0. Otherwise the stats for this table may stay 0 for ever before user load new to this table.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #30096:
URL: https://github.com/apache/doris/pull/30096#issuecomment-1897890462
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30096:
URL: https://github.com/apache/doris/pull/30096#issuecomment-1899544487
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30096:
URL: https://github.com/apache/doris/pull/30096#issuecomment-1897922248
<details>
<summary>TPC-H: <b>Total hot run time: 38926 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 359da989fd6b9516da8ac2a4dead0b919666b29d, data reload: false
------ Round 1 ----------------------------------
q1 17697 5796 5332 5332
q2 2026 145 135 135
q3 10769 1147 1158 1147
q4 10368 801 873 801
q5 7735 3156 3098 3098
q6 202 119 121 119
q7 849 501 484 484
q8 9252 1937 1979 1937
q9 7293 6347 6334 6334
q10 8184 3033 3060 3033
q11 406 212 211 211
q12 358 193 189 189
q13 18005 3391 3349 3349
q14 248 214 220 214
q15 544 532 504 504
q16 439 376 377 376
q17 970 542 519 519
q18 7613 6921 6814 6814
q19 1558 1374 1366 1366
q20 573 293 310 293
q21 2760 2386 2433 2386
q22 344 285 308 285
Total cold run time: 108193 ms
Total hot run time: 38926 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5534 5406 5388 5388
q2 330 217 218 217
q3 3351 3236 3228 3228
q4 2109 2054 2090 2054
q5 5948 5869 5822 5822
q6 206 119 120 119
q7 2343 1898 1862 1862
q8 3217 3399 3404 3399
q9 8880 8950 8863 8863
q10 3901 3854 3846 3846
q11 569 452 465 452
q12 812 630 648 630
q13 16899 3152 3139 3139
q14 287 259 273 259
q15 560 501 505 501
q16 511 480 471 471
q17 1919 1838 1825 1825
q18 9468 17401 9719 9719
q19 26465 1581 1535 1535
q20 4575 1925 1938 1925
q21 14667 5768 5456 5456
q22 975 551 544 544
Total cold run time: 113526 ms
Total hot run time: 61254 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #30096:
URL: https://github.com/apache/doris/pull/30096
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30096:
URL: https://github.com/apache/doris/pull/30096#issuecomment-1899544514
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30096:
URL: https://github.com/apache/doris/pull/30096#issuecomment-1897942550
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 359da989fd6b9516da8ac2a4dead0b919666b29d with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.0 seconds inserted 10000000 Rows, about 714K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30096:
URL: https://github.com/apache/doris/pull/30096#issuecomment-1897933715
<details>
<summary>TPC-DS: <b>Total hot run time: 176706 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 359da989fd6b9516da8ac2a4dead0b919666b29d, data reload: false
query1 954 334 330 330
query2 6562 1942 1901 1901
query3 6702 198 197 197
query4 32685 22088 22217 22088
query5 6885 598 611 598
query6 255 171 162 162
query7 4602 263 264 263
query8 229 170 179 170
query9 8359 2544 2527 2527
query10 427 228 224 224
query11 16840 15589 15515 15515
query12 124 68 69 68
query13 1686 381 372 372
query14 10491 7032 6901 6901
query15 199 184 200 184
query16 4626 232 236 232
query17 984 485 484 484
query18 1779 257 255 255
query19 178 135 128 128
query20 70 67 69 67
query21 192 124 137 124
query22 4887 4631 4779 4631
query23 32005 30933 30803 30803
query24 11783 2826 2793 2793
query25 569 316 304 304
query26 1611 140 141 140
query27 3220 272 270 270
query28 7152 1835 1842 1835
query29 1374 625 621 621
query30 279 140 138 138
query31 930 731 734 731
query32 83 52 47 47
query33 692 214 209 209
query34 1149 447 442 442
query35 887 780 761 761
query36 1329 1170 1207 1170
query37 96 57 56 56
query38 3382 3198 3174 3174
query39 1323 1279 1262 1262
query40 211 81 86 81
query41 37 34 34 34
query42 87 84 81 81
query43 505 487 482 482
query44 1047 692 691 691
query45 191 181 175 175
query46 1063 657 650 650
query47 1595 1560 1517 1517
query48 389 305 322 305
query49 1126 283 285 283
query50 676 313 318 313
query51 5286 5160 5191 5160
query52 101 72 71 71
query53 327 259 254 254
query54 890 434 441 434
query55 87 76 74 74
query56 171 163 168 163
query57 991 943 925 925
query58 190 158 154 154
query59 2979 2770 2711 2711
query60 215 181 182 181
query61 83 82 81 81
query62 629 367 350 350
query63 277 260 260 260
query64 4946 1750 1721 1721
query65 3326 3250 3229 3229
query66 1265 325 311 311
query67 15556 15164 15409 15164
query68 10961 536 525 525
query69 601 293 304 293
query70 1725 1547 1484 1484
query71 553 207 202 202
query72 5004 2832 2828 2828
query73 2098 309 310 309
query74 6938 6434 6442 6434
query75 4919 2327 2265 2265
query76 6234 1062 1032 1032
query77 675 233 238 233
query78 9024 8866 8577 8577
query79 4375 505 511 505
query80 707 326 315 315
query81 476 209 212 209
query82 202 78 85 78
query83 136 117 119 117
query84 269 65 69 65
query85 1065 333 315 315
query86 400 398 390 390
query87 3479 3350 3333 3333
query88 3361 2149 2153 2149
query89 446 348 353 348
query90 2002 183 184 183
query91 178 122 133 122
query92 55 42 43 42
query93 3007 467 420 420
query94 1165 159 156 156
query95 491 458 438 438
query96 612 315 314 314
query97 4270 4175 4150 4150
query98 199 197 187 187
query99 1017 728 656 656
Total cold run time: 298470 ms
Total hot run time: 176706 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](statistics)Reanalyze olapTable if getRowCount is not 0 and last time row count is 0. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30096:
URL: https://github.com/apache/doris/pull/30096#issuecomment-1897939274
<details>
<summary>ClickBench: <b>Total hot run time: 30.64 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 359da989fd6b9516da8ac2a4dead0b919666b29d, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.02 0.02
query3 0.23 0.05 0.05
query4 1.67 0.08 0.06
query5 0.53 0.53 0.52
query6 1.29 0.63 0.64
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.55 0.51 0.50
query10 0.56 0.55 0.55
query11 0.12 0.09 0.08
query12 0.11 0.09 0.09
query13 0.60 0.59 0.60
query14 0.77 0.77 0.80
query15 0.80 0.78 0.77
query16 0.37 0.35 0.39
query17 1.04 1.01 1.03
query18 0.25 0.23 0.27
query19 1.87 1.82 1.79
query20 0.01 0.01 0.01
query21 15.40 0.58 0.57
query22 2.59 2.44 2.16
query23 17.09 0.84 0.82
query24 2.75 0.50 1.45
query25 0.40 0.26 0.14
query26 0.51 0.13 0.13
query27 0.06 0.06 0.05
query28 11.32 0.78 0.77
query29 12.51 3.11 3.12
query30 0.53 0.52 0.49
query31 2.78 0.35 0.34
query32 3.37 0.47 0.47
query33 3.26 3.21 3.24
query34 15.76 4.25 4.27
query35 4.25 4.22 4.24
query36 1.10 1.08 1.08
query37 0.07 0.05 0.05
query38 0.04 0.03 0.02
query39 0.02 0.01 0.02
query40 0.15 0.13 0.13
query41 0.07 0.01 0.01
query42 0.02 0.02 0.01
query43 0.03 0.01 0.02
Total cold run time: 105.02 s
Total hot run time: 30.64 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org