You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2024/10/24 08:36:47 UTC
[PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Jibing-Li opened a new pull request, #42399:
URL: https://github.com/apache/doris/pull/42399
Support auto analyze columns that haven't been analyzed for a long time.
Add another thread to find the tables which are not analyzed for a long time and append it to a job queue. The priority of this kind of jobs are the the lowest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2453964686
<details>
<summary>ClickBench: <b>Total hot run time: 31.97 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a89d592b48e8286fa1d33f582b1486a07e71c877, data reload: false
query1 0.03 0.03 0.03
query2 0.07 0.03 0.03
query3 0.23 0.06 0.06
query4 1.64 0.10 0.10
query5 0.41 0.42 0.43
query6 1.14 0.66 0.64
query7 0.02 0.02 0.02
query8 0.04 0.03 0.04
query9 0.57 0.49 0.49
query10 0.55 0.55 0.54
query11 0.13 0.10 0.10
query12 0.13 0.11 0.10
query13 0.61 0.60 0.60
query14 2.72 2.73 2.76
query15 0.89 0.82 0.82
query16 0.39 0.39 0.37
query17 0.99 1.00 1.04
query18 0.20 0.19 0.20
query19 1.94 1.89 1.95
query20 0.01 0.01 0.01
query21 15.37 0.60 0.59
query22 2.28 1.89 1.44
query23 17.00 1.05 0.64
query24 3.12 0.99 1.27
query25 0.45 0.20 0.03
query26 0.25 0.14 0.15
query27 0.04 0.05 0.04
query28 10.62 1.09 1.06
query29 12.57 3.25 3.23
query30 0.24 0.06 0.05
query31 2.87 0.40 0.39
query32 3.27 0.47 0.46
query33 2.96 3.03 3.05
query34 17.20 4.45 4.44
query35 4.48 4.54 4.49
query36 0.68 0.48 0.49
query37 0.10 0.06 0.06
query38 0.04 0.03 0.03
query39 0.03 0.02 0.03
query40 0.16 0.12 0.12
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.04 0.03 0.04
Total cold run time: 106.6 s
Total hot run time: 31.97 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li merged PR #42399:
URL: https://github.com/apache/doris/pull/42399
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2437125163
<details>
<summary>TPC-H: <b>Total hot run time: 41053 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b57e04b68aee2b2894c876929c834d47b38e26b, data reload: false
------ Round 1 ----------------------------------
q1 17600 7387 7268 7268
q2 2013 276 271 271
q3 11993 1059 1211 1059
q4 10571 860 904 860
q5 7741 3107 3043 3043
q6 233 150 145 145
q7 1010 607 588 588
q8 9366 1966 1993 1966
q9 6533 6436 6423 6423
q10 7036 2483 2399 2399
q11 443 249 249 249
q12 398 214 210 210
q13 17767 2986 2989 2986
q14 255 205 219 205
q15 561 520 509 509
q16 646 584 592 584
q17 954 526 560 526
q18 7260 6693 6607 6607
q19 1341 1003 966 966
q20 492 184 180 180
q21 3944 3280 2992 2992
q22 1119 1029 1017 1017
Total cold run time: 109276 ms
Total hot run time: 41053 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7244 7224 7243 7224
q2 324 234 240 234
q3 2997 3000 2923 2923
q4 2077 1854 1791 1791
q5 5774 5742 5794 5742
q6 231 144 139 139
q7 2224 1847 1783 1783
q8 3378 3533 3443 3443
q9 8882 8924 8861 8861
q10 3593 3567 3588 3567
q11 581 492 476 476
q12 865 641 592 592
q13 9083 3167 3193 3167
q14 311 283 266 266
q15 583 519 519 519
q16 689 643 649 643
q17 1840 1611 1601 1601
q18 8298 7870 7786 7786
q19 1712 1629 1630 1629
q20 2149 1883 1863 1863
q21 5618 5489 5476 5476
q22 1179 1045 1034 1034
Total cold run time: 69632 ms
Total hot run time: 60759 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2528034233
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2434645264
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2453957784
<details>
<summary>TPC-DS: <b>Total hot run time: 191814 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a89d592b48e8286fa1d33f582b1486a07e71c877, data reload: false
query1 965 385 392 385
query2 6506 2095 2072 2072
query3 6780 222 215 215
query4 33931 23887 23634 23634
query5 4353 455 448 448
query6 274 196 176 176
query7 4615 312 336 312
query8 313 218 230 218
query9 9498 2816 2803 2803
query10 480 255 244 244
query11 18142 15434 15177 15177
query12 151 101 101 101
query13 1672 449 432 432
query14 10314 6440 6387 6387
query15 243 180 178 178
query16 7991 444 468 444
query17 1604 595 556 556
query18 2135 309 309 309
query19 245 144 143 143
query20 117 109 112 109
query21 207 102 102 102
query22 4625 4165 4242 4165
query23 35055 34099 34249 34099
query24 11564 2734 2789 2734
query25 696 398 393 393
query26 1832 162 163 162
query27 2907 291 301 291
query28 8360 2644 2620 2620
query29 1049 431 426 426
query30 334 161 150 150
query31 1039 801 811 801
query32 96 59 60 59
query33 778 290 287 287
query34 977 512 521 512
query35 908 738 730 730
query36 1093 946 968 946
query37 202 75 83 75
query38 4388 4344 4344 4344
query39 1477 1432 1424 1424
query40 290 107 106 106
query41 52 46 46 46
query42 112 98 103 98
query43 537 502 491 491
query44 1351 910 892 892
query45 185 164 165 164
query46 1143 714 704 704
query47 1949 1825 1844 1825
query48 416 329 323 323
query49 1301 404 396 396
query50 810 397 391 391
query51 7453 7267 7158 7158
query52 104 94 89 89
query53 257 187 181 181
query54 1149 427 439 427
query55 81 80 84 80
query56 270 268 251 251
query57 1319 1220 1156 1156
query58 230 195 212 195
query59 3356 3236 3103 3103
query60 275 242 252 242
query61 111 100 107 100
query62 884 674 692 674
query63 218 185 183 183
query64 5211 682 640 640
query65 3363 3246 3189 3189
query66 1416 300 302 300
query67 16465 15735 15605 15605
query68 4716 662 648 648
query69 439 247 251 247
query70 1316 1118 1108 1108
query71 419 259 285 259
query72 6406 3980 4006 3980
query73 774 351 355 351
query74 10361 8993 9004 8993
query75 3462 2662 2700 2662
query76 2885 1020 1079 1020
query77 393 289 280 280
query78 10338 9408 9394 9394
query79 1784 623 648 623
query80 1096 467 438 438
query81 548 237 238 237
query82 926 117 124 117
query83 214 135 135 135
query84 237 71 67 67
query85 1245 322 297 297
query86 362 313 290 290
query87 4845 4692 4674 4674
query88 3258 2175 2132 2132
query89 404 291 289 289
query90 1994 182 183 182
query91 132 101 100 100
query92 54 50 49 49
query93 1286 598 590 590
query94 911 293 296 293
query95 345 250 247 247
query96 615 282 280 280
query97 2899 2672 2698 2672
query98 211 197 195 195
query99 1552 1295 1293 1293
Total cold run time: 304651 ms
Total hot run time: 191814 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2527738442
<details>
<summary>TPC-DS: <b>Total hot run time: 197233 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9cb6827dd47fde8bd54e1ecd5dd2aa86454a5a66, data reload: false
query1 1246 945 943 943
query2 6248 2031 2022 2022
query3 10960 4428 4343 4343
query4 67549 29160 23412 23412
query5 4894 481 462 462
query6 425 196 194 194
query7 5552 310 295 295
query8 326 244 237 237
query9 8741 2736 2711 2711
query10 452 243 241 241
query11 17202 15379 15878 15379
query12 157 103 105 103
query13 1458 443 411 411
query14 10924 7772 7358 7358
query15 215 186 197 186
query16 7147 499 464 464
query17 1140 623 559 559
query18 1782 303 311 303
query19 205 152 157 152
query20 117 113 115 113
query21 208 104 105 104
query22 4717 4458 4559 4458
query23 34697 34462 34636 34462
query24 5533 2478 2494 2478
query25 493 386 385 385
query26 656 157 161 157
query27 1804 281 284 281
query28 4461 2509 2493 2493
query29 673 415 430 415
query30 224 155 157 155
query31 1003 809 852 809
query32 68 56 65 56
query33 446 301 315 301
query34 968 520 548 520
query35 866 775 764 764
query36 1115 966 988 966
query37 124 83 80 80
query38 4487 4382 4392 4382
query39 1530 1477 1482 1477
query40 210 98 97 97
query41 44 41 45 41
query42 116 102 97 97
query43 535 502 504 502
query44 1245 858 870 858
query45 198 171 168 168
query46 1166 753 717 717
query47 2029 1942 1926 1926
query48 432 322 319 319
query49 735 399 402 399
query50 860 399 409 399
query51 7378 7171 7070 7070
query52 99 87 85 85
query53 263 186 180 180
query54 508 396 397 396
query55 79 77 78 77
query56 244 236 253 236
query57 1267 1142 1118 1118
query58 226 232 227 227
query59 3087 2973 2934 2934
query60 306 283 263 263
query61 137 128 128 128
query62 812 661 664 661
query63 213 229 184 184
query64 1354 645 636 636
query65 3256 3213 3207 3207
query66 701 298 297 297
query67 15815 15583 15685 15583
query68 4007 580 578 578
query69 427 267 245 245
query70 1172 1146 1165 1146
query71 359 252 248 248
query72 6367 4074 3746 3746
query73 761 359 362 359
query74 10425 9123 8916 8916
query75 3505 2717 2659 2659
query76 1782 1223 1068 1068
query77 487 268 270 268
query78 10623 9492 9457 9457
query79 2214 611 618 611
query80 1372 452 431 431
query81 504 235 236 235
query82 1242 120 125 120
query83 171 139 142 139
query84 279 72 71 71
query85 960 299 292 292
query86 415 310 298 298
query87 4939 4541 4622 4541
query88 3604 2235 2192 2192
query89 434 319 298 298
query90 1635 194 188 188
query91 138 104 100 100
query92 102 50 52 50
query93 2909 554 542 542
query94 698 291 289 289
query95 346 263 255 255
query96 640 284 282 282
query97 2860 2680 2667 2667
query98 223 200 201 200
query99 1608 1306 1320 1306
Total cold run time: 321336 ms
Total hot run time: 197233 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2434736103
<details>
<summary>ClickBench: <b>Total hot run time: 32.48 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccc3ae1721f8241c4724680b176424edc833a678, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.03 0.03
query3 0.22 0.06 0.06
query4 1.65 0.10 0.10
query5 0.42 0.41 0.40
query6 1.17 0.66 0.65
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.57 0.50 0.52
query10 0.56 0.56 0.55
query11 0.13 0.10 0.11
query12 0.15 0.11 0.11
query13 0.61 0.59 0.59
query14 2.68 2.80 2.77
query15 0.90 0.84 0.83
query16 0.40 0.40 0.39
query17 1.01 1.02 1.05
query18 0.23 0.22 0.22
query19 1.98 1.87 2.02
query20 0.01 0.01 0.01
query21 15.36 0.58 0.58
query22 2.69 1.94 2.05
query23 17.05 1.02 0.72
query24 3.17 1.08 0.83
query25 0.26 0.08 0.11
query26 0.38 0.14 0.13
query27 0.04 0.04 0.04
query28 10.86 1.09 1.07
query29 12.57 3.30 3.24
query30 0.24 0.06 0.06
query31 2.87 0.39 0.37
query32 3.26 0.46 0.45
query33 3.00 3.02 3.03
query34 16.91 4.48 4.42
query35 4.47 4.54 4.49
query36 0.67 0.47 0.51
query37 0.08 0.06 0.06
query38 0.05 0.03 0.04
query39 0.03 0.02 0.03
query40 0.17 0.12 0.12
query41 0.08 0.02 0.02
query42 0.03 0.03 0.02
query43 0.03 0.03 0.03
Total cold run time: 107.12 s
Total hot run time: 32.48 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2530224360
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2527620990
<details>
<summary>ClickBench: <b>Total hot run time: 33.3 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4fde56054bffbc8e3900284c2a25b03486e2c177, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.04 0.03
query3 0.23 0.07 0.07
query4 1.63 0.10 0.10
query5 0.46 0.39 0.40
query6 1.14 0.66 0.65
query7 0.02 0.02 0.02
query8 0.04 0.04 0.04
query9 0.57 0.53 0.50
query10 0.55 0.54 0.54
query11 0.14 0.10 0.11
query12 0.14 0.11 0.12
query13 0.61 0.61 0.62
query14 2.81 2.74 2.74
query15 0.90 0.83 0.83
query16 0.39 0.38 0.38
query17 1.00 1.10 1.05
query18 0.23 0.21 0.22
query19 2.00 1.73 2.03
query20 0.01 0.01 0.01
query21 15.41 0.59 0.55
query22 2.70 1.88 2.40
query23 17.03 1.18 0.82
query24 2.48 1.75 2.03
query25 0.42 0.19 0.08
query26 0.50 0.14 0.14
query27 0.04 0.05 0.04
query28 9.93 1.11 1.08
query29 12.56 3.24 3.25
query30 0.25 0.06 0.07
query31 2.88 0.39 0.39
query32 3.23 0.48 0.47
query33 3.00 3.06 3.04
query34 17.05 4.41 4.46
query35 4.51 4.43 4.49
query36 0.69 0.50 0.51
query37 0.09 0.06 0.05
query38 0.05 0.03 0.03
query39 0.03 0.02 0.03
query40 0.18 0.12 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 106.14 s
Total hot run time: 33.3 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2527758090
<details>
<summary>ClickBench: <b>Total hot run time: 32.81 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9cb6827dd47fde8bd54e1ecd5dd2aa86454a5a66, data reload: false
query1 0.03 0.02 0.04
query2 0.07 0.03 0.04
query3 0.23 0.07 0.08
query4 1.62 0.10 0.10
query5 0.42 0.42 0.39
query6 1.18 0.67 0.65
query7 0.02 0.02 0.02
query8 0.04 0.03 0.03
query9 0.58 0.51 0.48
query10 0.55 0.59 0.56
query11 0.14 0.10 0.10
query12 0.14 0.11 0.12
query13 0.61 0.60 0.59
query14 2.72 2.73 2.80
query15 0.90 0.83 0.82
query16 0.38 0.37 0.39
query17 0.99 1.05 0.98
query18 0.23 0.20 0.21
query19 1.84 1.86 1.97
query20 0.02 0.01 0.01
query21 15.38 0.58 0.58
query22 2.59 2.56 1.94
query23 16.94 1.03 0.87
query24 3.27 1.25 1.05
query25 0.21 0.19 0.19
query26 0.51 0.13 0.13
query27 0.04 0.04 0.04
query28 10.46 1.10 1.07
query29 12.57 3.20 3.20
query30 0.24 0.07 0.06
query31 2.88 0.38 0.38
query32 3.26 0.45 0.46
query33 2.99 3.10 3.04
query34 16.99 4.43 4.44
query35 4.46 4.49 4.49
query36 0.66 0.50 0.48
query37 0.10 0.06 0.06
query38 0.05 0.03 0.04
query39 0.03 0.02 0.03
query40 0.17 0.12 0.13
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.03 0.03 0.02
Total cold run time: 106.66 s
Total hot run time: 32.81 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2437158477
<details>
<summary>ClickBench: <b>Total hot run time: 32.42 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b57e04b68aee2b2894c876929c834d47b38e26b, data reload: false
query1 0.04 0.04 0.03
query2 0.07 0.03 0.03
query3 0.23 0.06 0.07
query4 1.64 0.10 0.10
query5 0.42 0.39 0.40
query6 1.14 0.65 0.65
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.57 0.50 0.50
query10 0.56 0.55 0.54
query11 0.15 0.10 0.10
query12 0.13 0.11 0.11
query13 0.60 0.59 0.59
query14 2.88 2.81 2.73
query15 0.89 0.82 0.84
query16 0.39 0.39 0.40
query17 1.06 1.04 1.04
query18 0.20 0.20 0.20
query19 1.90 1.88 2.00
query20 0.02 0.01 0.01
query21 15.35 0.57 0.58
query22 2.84 3.25 1.73
query23 17.06 0.99 0.85
query24 3.29 1.24 0.96
query25 0.13 0.25 0.04
query26 0.52 0.14 0.13
query27 0.05 0.04 0.03
query28 10.27 1.11 1.07
query29 12.56 3.30 3.29
query30 0.25 0.06 0.06
query31 2.86 0.38 0.37
query32 3.28 0.47 0.45
query33 2.98 3.01 3.00
query34 17.12 4.49 4.45
query35 4.44 4.46 4.47
query36 0.66 0.47 0.49
query37 0.08 0.06 0.05
query38 0.04 0.04 0.03
query39 0.04 0.02 0.02
query40 0.17 0.12 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.02 0.03
Total cold run time: 107.08 s
Total hot run time: 32.42 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2434723989
<details>
<summary>TPC-DS: <b>Total hot run time: 192221 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccc3ae1721f8241c4724680b176424edc833a678, data reload: false
query1 980 378 369 369
query2 6526 2075 2083 2075
query3 6790 211 221 211
query4 34379 23614 23499 23499
query5 4383 474 482 474
query6 266 175 184 175
query7 4607 298 293 293
query8 281 240 236 236
query9 9544 2715 2728 2715
query10 480 299 275 275
query11 18080 15327 15216 15216
query12 158 103 101 101
query13 1693 439 405 405
query14 10416 7134 7260 7134
query15 283 174 186 174
query16 8120 458 461 458
query17 1665 578 557 557
query18 2139 297 297 297
query19 364 151 150 150
query20 119 108 106 106
query21 218 107 103 103
query22 4664 4463 4400 4400
query23 34931 34183 33882 33882
query24 11191 2759 2779 2759
query25 679 403 416 403
query26 1377 160 160 160
query27 2833 281 288 281
query28 7949 2464 2460 2460
query29 930 423 428 423
query30 338 166 161 161
query31 1043 830 829 829
query32 99 59 57 57
query33 776 290 301 290
query34 940 516 524 516
query35 1067 889 908 889
query36 1080 969 961 961
query37 158 98 92 92
query38 4389 4291 4366 4291
query39 1465 1416 1403 1403
query40 287 102 99 99
query41 50 46 47 46
query42 128 101 98 98
query43 540 485 486 485
query44 1297 814 807 807
query45 199 165 168 165
query46 1144 690 685 685
query47 1950 1838 1857 1838
query48 427 319 318 318
query49 1173 444 473 444
query50 804 381 393 381
query51 7149 6989 6950 6950
query52 101 92 90 90
query53 254 186 180 180
query54 1240 444 429 429
query55 79 80 81 80
query56 288 277 289 277
query57 1362 1179 1163 1163
query58 260 245 290 245
query59 3218 3040 3148 3040
query60 303 260 262 260
query61 104 101 100 100
query62 877 667 681 667
query63 216 190 183 183
query64 5308 643 607 607
query65 3271 3267 3224 3224
query66 1454 315 320 315
query67 16122 15748 15731 15731
query68 4857 569 550 550
query69 454 317 304 304
query70 1131 1153 1111 1111
query71 337 277 275 275
query72 6361 3975 3918 3918
query73 776 355 360 355
query74 10073 8962 8953 8953
query75 3428 2709 2704 2704
query76 2810 890 916 890
query77 415 295 310 295
query78 10658 9643 9510 9510
query79 1477 592 601 592
query80 1748 444 437 437
query81 574 242 254 242
query82 1220 138 151 138
query83 252 146 136 136
query84 263 67 69 67
query85 1371 300 284 284
query86 387 303 309 303
query87 4750 4638 4693 4638
query88 3538 2212 2191 2191
query89 396 288 292 288
query90 2046 189 184 184
query91 134 100 105 100
query92 71 50 49 49
query93 1035 528 533 528
query94 1101 295 302 295
query95 358 251 246 246
query96 616 278 283 278
query97 2888 2678 2694 2678
query98 207 195 206 195
query99 1649 1300 1300 1300
Total cold run time: 304386 ms
Total hot run time: 192221 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2437071150
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2527608805
<details>
<summary>TPC-DS: <b>Total hot run time: 197674 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4fde56054bffbc8e3900284c2a25b03486e2c177, data reload: false
query1 1273 963 986 963
query2 6234 2133 2043 2043
query3 10959 4411 4411 4411
query4 67346 28623 23490 23490
query5 4959 465 471 465
query6 424 187 184 184
query7 5587 313 299 299
query8 323 240 236 236
query9 9253 2739 2719 2719
query10 442 255 238 238
query11 17374 15265 15795 15265
query12 144 113 107 107
query13 1546 425 414 414
query14 10909 7349 7269 7269
query15 215 183 208 183
query16 7514 506 482 482
query17 1066 590 596 590
query18 1657 310 315 310
query19 209 161 158 158
query20 121 114 111 111
query21 221 101 105 101
query22 4727 4473 4678 4473
query23 34882 34832 34480 34480
query24 5406 2597 2465 2465
query25 489 391 389 389
query26 666 154 156 154
query27 1903 300 294 294
query28 4463 2502 2463 2463
query29 681 471 411 411
query30 205 153 154 153
query31 1023 832 828 828
query32 64 53 53 53
query33 446 287 288 287
query34 938 515 547 515
query35 851 782 764 764
query36 1108 969 959 959
query37 123 71 75 71
query38 4518 4439 4616 4439
query39 1521 1459 1459 1459
query40 203 106 97 97
query41 45 43 42 42
query42 109 102 102 102
query43 543 485 481 481
query44 1208 846 849 846
query45 193 168 166 166
query46 1166 737 725 725
query47 2074 1978 1918 1918
query48 434 324 328 324
query49 730 391 394 391
query50 849 388 416 388
query51 7341 7096 7183 7096
query52 98 90 86 86
query53 259 185 183 183
query54 507 406 388 388
query55 80 78 79 78
query56 249 248 250 248
query57 1253 1094 1122 1094
query58 214 206 210 206
query59 3262 3005 2946 2946
query60 264 243 249 243
query61 109 150 102 102
query62 773 679 662 662
query63 215 184 188 184
query64 1398 679 629 629
query65 3305 3177 3221 3177
query66 757 297 288 288
query67 16335 15642 15681 15642
query68 4004 589 561 561
query69 431 266 256 256
query70 1206 1176 1155 1155
query71 359 252 246 246
query72 6458 4088 4119 4088
query73 771 368 358 358
query74 9509 9283 9052 9052
query75 3398 2635 2660 2635
query76 1780 1117 1089 1089
query77 473 270 264 264
query78 10409 9520 9422 9422
query79 2291 620 621 620
query80 1439 433 429 429
query81 509 231 235 231
query82 1287 122 111 111
query83 165 157 138 138
query84 290 74 74 74
query85 1005 304 289 289
query86 413 280 298 280
query87 4700 4666 4604 4604
query88 3424 2233 2178 2178
query89 422 295 304 295
query90 1962 189 187 187
query91 142 103 101 101
query92 63 50 50 50
query93 2860 548 547 547
query94 881 296 293 293
query95 348 266 249 249
query96 625 278 276 276
query97 2860 2670 2678 2670
query98 216 198 194 194
query99 1635 1308 1305 1305
Total cold run time: 322098 ms
Total hot run time: 197674 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2488455367
<details>
<summary>TPC-DS: <b>Total hot run time: 195861 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 990467635c439b25faca646c03206c93bb16e9f7, data reload: false
query1 1248 944 968 944
query2 6268 2212 2224 2212
query3 10801 4003 3912 3912
query4 67599 28166 23662 23662
query5 4973 463 444 444
query6 398 183 188 183
query7 5518 293 281 281
query8 295 218 220 218
query9 8430 2644 2631 2631
query10 446 240 233 233
query11 17393 15250 15772 15250
query12 150 99 106 99
query13 1465 429 428 428
query14 11101 6903 7358 6903
query15 219 181 193 181
query16 6986 474 451 451
query17 1016 600 546 546
query18 1795 282 284 282
query19 196 153 149 149
query20 114 109 111 109
query21 198 105 99 99
query22 4640 4418 4392 4392
query23 35464 34129 34075 34075
query24 5371 2400 2453 2400
query25 488 374 375 374
query26 631 173 153 153
query27 1663 281 283 281
query28 4223 2472 2401 2401
query29 645 405 405 405
query30 210 145 144 144
query31 989 761 823 761
query32 66 54 57 54
query33 436 278 283 278
query34 930 489 511 489
query35 862 715 736 715
query36 1094 952 975 952
query37 118 76 70 70
query38 4374 4254 4197 4197
query39 1466 1447 1445 1445
query40 199 99 99 99
query41 44 42 42 42
query42 108 98 92 92
query43 541 512 510 510
query44 1162 783 815 783
query45 183 164 160 160
query46 1129 685 698 685
query47 1950 1845 1810 1810
query48 415 304 321 304
query49 737 401 400 400
query50 814 397 410 397
query51 7353 7070 7206 7070
query52 98 84 92 84
query53 258 179 178 178
query54 517 391 389 389
query55 80 77 80 77
query56 247 233 236 233
query57 1290 1184 1173 1173
query58 228 231 217 217
query59 3366 3091 3200 3091
query60 277 266 249 249
query61 137 142 135 135
query62 792 670 686 670
query63 216 195 183 183
query64 1500 784 747 747
query65 3283 3254 3234 3234
query66 720 325 377 325
query67 15981 15721 15665 15665
query68 3549 575 566 566
query69 426 252 255 252
query70 1203 1152 1138 1138
query71 389 242 254 242
query72 6223 4098 4022 4022
query73 779 367 360 360
query74 9889 8967 9010 8967
query75 3402 2671 2667 2667
query76 1871 1056 1015 1015
query77 518 300 289 289
query78 10585 9614 9410 9410
query79 1459 608 603 603
query80 836 497 437 437
query81 528 230 229 229
query82 1307 120 113 113
query83 212 154 202 154
query84 290 72 74 72
query85 894 297 307 297
query86 338 308 308 308
query87 4781 4636 4556 4556
query88 3541 2199 2171 2171
query89 435 295 296 295
query90 1996 190 188 188
query91 136 101 105 101
query92 65 50 51 50
query93 1964 548 545 545
query94 755 307 287 287
query95 357 252 250 250
query96 631 284 295 284
query97 2918 2709 2690 2690
query98 212 200 200 200
query99 1630 1311 1298 1298
Total cold run time: 318023 ms
Total hot run time: 195861 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2453872744
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2454117771
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2488342957
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2488427793
<details>
<summary>TPC-H: <b>Total hot run time: 39765 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 990467635c439b25faca646c03206c93bb16e9f7, data reload: false
------ Round 1 ----------------------------------
q1 17589 7521 7317 7317
q2 2058 176 165 165
q3 10603 1080 1248 1080
q4 10576 703 698 698
q5 7592 2794 2713 2713
q6 243 147 144 144
q7 981 619 607 607
q8 9260 1878 1926 1878
q9 6630 6380 6588 6380
q10 7046 2283 2328 2283
q11 459 261 263 261
q12 427 215 212 212
q13 17772 2989 3016 2989
q14 252 207 218 207
q15 567 534 526 526
q16 698 577 573 573
q17 976 561 534 534
q18 7271 6639 6805 6639
q19 1347 1043 913 913
q20 457 182 179 179
q21 4254 3278 3152 3152
q22 373 320 315 315
Total cold run time: 107431 ms
Total hot run time: 39765 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7260 7299 7330 7299
q2 330 229 228 228
q3 2931 2849 2982 2849
q4 2047 1800 1874 1800
q5 5677 5679 5686 5679
q6 236 143 145 143
q7 2260 1840 1848 1840
q8 3481 3554 3545 3545
q9 8895 8898 8900 8898
q10 3605 3592 3584 3584
q11 599 515 512 512
q12 834 634 634 634
q13 13320 3215 3232 3215
q14 305 281 279 279
q15 575 528 544 528
q16 734 647 659 647
q17 1839 1661 1609 1609
q18 8393 7891 7635 7635
q19 1729 1623 1536 1536
q20 2123 1882 1887 1882
q21 5699 5375 5397 5375
q22 664 562 577 562
Total cold run time: 73536 ms
Total hot run time: 60279 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2488466697
<details>
<summary>ClickBench: <b>Total hot run time: 32.67 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 990467635c439b25faca646c03206c93bb16e9f7, data reload: false
query1 0.04 0.03 0.02
query2 0.07 0.03 0.04
query3 0.24 0.08 0.07
query4 1.61 0.10 0.10
query5 0.42 0.41 0.39
query6 1.15 0.66 0.67
query7 0.02 0.01 0.01
query8 0.04 0.02 0.03
query9 0.59 0.50 0.50
query10 0.55 0.55 0.56
query11 0.14 0.11 0.10
query12 0.13 0.11 0.11
query13 0.61 0.60 0.60
query14 2.85 2.75 2.76
query15 0.90 0.84 0.82
query16 0.38 0.39 0.40
query17 1.03 1.08 1.06
query18 0.21 0.20 0.20
query19 1.98 1.88 1.95
query20 0.01 0.01 0.01
query21 15.49 0.59 0.56
query22 2.29 2.36 2.21
query23 17.11 1.11 0.81
query24 2.98 1.15 0.58
query25 0.30 0.11 0.10
query26 0.38 0.14 0.13
query27 0.06 0.04 0.04
query28 10.90 1.11 1.08
query29 12.64 3.24 3.18
query30 0.25 0.07 0.07
query31 2.87 0.40 0.37
query32 3.29 0.47 0.47
query33 2.93 3.04 3.09
query34 16.80 4.48 4.47
query35 4.48 4.49 4.53
query36 0.68 0.50 0.48
query37 0.09 0.06 0.06
query38 0.06 0.04 0.04
query39 0.04 0.02 0.02
query40 0.16 0.13 0.13
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.03 0.04 0.03
Total cold run time: 106.92 s
Total hot run time: 32.67 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2527580491
<details>
<summary>TPC-H: <b>Total hot run time: 39711 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4fde56054bffbc8e3900284c2a25b03486e2c177, data reload: false
------ Round 1 ----------------------------------
q1 17625 7453 7232 7232
q2 2031 173 165 165
q3 10733 1069 1144 1069
q4 10527 774 736 736
q5 7605 2706 2711 2706
q6 233 148 149 148
q7 1015 644 600 600
q8 9256 1804 1926 1804
q9 6666 6438 6411 6411
q10 7009 2287 2291 2287
q11 465 264 258 258
q12 422 219 227 219
q13 17790 3059 3022 3022
q14 233 211 206 206
q15 569 535 520 520
q16 669 581 594 581
q17 964 565 586 565
q18 7254 6787 6484 6484
q19 1345 977 1064 977
q20 477 179 177 177
q21 3950 3232 3232 3232
q22 388 320 312 312
Total cold run time: 107226 ms
Total hot run time: 39711 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7212 7268 7218 7218
q2 336 237 230 230
q3 2915 2764 2909 2764
q4 2113 1811 1771 1771
q5 5612 5645 5639 5639
q6 232 146 142 142
q7 2296 1825 1857 1825
q8 3350 3569 3493 3493
q9 8951 9131 8969 8969
q10 3611 3519 3548 3519
q11 605 513 560 513
q12 789 609 609 609
q13 12476 3221 3279 3221
q14 307 275 267 267
q15 568 525 518 518
q16 714 655 644 644
q17 1830 1600 1563 1563
q18 8173 7648 7575 7575
q19 1689 1567 1558 1558
q20 2106 1848 1865 1848
q21 5531 5428 5294 5294
q22 634 588 559 559
Total cold run time: 72050 ms
Total hot run time: 59739 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2437146565
<details>
<summary>TPC-DS: <b>Total hot run time: 192416 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b57e04b68aee2b2894c876929c834d47b38e26b, data reload: false
query1 866 390 401 390
query2 6245 2050 2027 2027
query3 8686 189 197 189
query4 35193 23663 23685 23663
query5 4831 479 433 433
query6 282 196 168 168
query7 4204 289 285 285
query8 285 237 244 237
query9 9684 2652 2651 2651
query10 496 247 241 241
query11 18138 15264 15215 15215
query12 149 107 102 102
query13 1573 454 400 400
query14 9666 7418 7425 7418
query15 249 176 179 176
query16 7927 455 488 455
query17 1535 598 612 598
query18 2143 317 305 305
query19 358 158 160 158
query20 127 114 113 113
query21 205 106 107 106
query22 5033 4486 4630 4486
query23 34878 34186 34115 34115
query24 10698 2839 2801 2801
query25 594 406 404 404
query26 718 159 168 159
query27 2026 280 282 280
query28 6222 2423 2382 2382
query29 739 414 423 414
query30 254 169 165 165
query31 1135 781 807 781
query32 104 55 57 55
query33 670 284 273 273
query34 908 509 510 509
query35 1052 898 863 863
query36 1121 935 952 935
query37 117 72 79 72
query38 4426 4199 4259 4199
query39 1484 1418 1452 1418
query40 200 99 101 99
query41 50 45 46 45
query42 112 96 93 93
query43 530 490 481 481
query44 1164 820 800 800
query45 183 175 169 169
query46 1125 721 700 700
query47 1941 1858 1859 1858
query48 403 323 339 323
query49 904 411 398 398
query50 814 393 394 393
query51 6995 7003 6891 6891
query52 109 87 85 85
query53 254 180 179 179
query54 1171 418 404 404
query55 79 76 76 76
query56 271 273 248 248
query57 1304 1217 1183 1183
query58 239 192 203 192
query59 3281 2802 3098 2802
query60 278 244 243 243
query61 100 104 107 104
query62 858 670 686 670
query63 214 185 178 178
query64 3851 626 615 615
query65 3273 3214 3215 3214
query66 862 304 305 304
query67 16041 15841 15900 15841
query68 4442 553 563 553
query69 429 255 260 255
query70 1177 1109 1137 1109
query71 330 264 263 263
query72 6311 4097 3918 3918
query73 774 364 360 360
query74 10315 9100 9035 9035
query75 3432 2646 2687 2646
query76 2825 1102 995 995
query77 382 267 271 267
query78 10614 9620 9583 9583
query79 2129 576 588 576
query80 1056 419 418 418
query81 558 241 242 241
query82 634 117 113 113
query83 240 133 132 132
query84 244 76 71 71
query85 1289 300 286 286
query86 433 287 286 286
query87 4890 4773 4764 4764
query88 3620 2183 2175 2175
query89 399 291 289 289
query90 2045 185 183 183
query91 135 101 98 98
query92 59 48 48 48
query93 2373 536 544 536
query94 936 291 255 255
query95 356 245 244 244
query96 620 283 289 283
query97 2848 2772 2710 2710
query98 214 205 201 201
query99 1564 1306 1294 1294
Total cold run time: 299300 ms
Total hot run time: 192416 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2434651871
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2453941834
<details>
<summary>TPC-H: <b>Total hot run time: 41620 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a89d592b48e8286fa1d33f582b1486a07e71c877, data reload: false
------ Round 1 ----------------------------------
q1 17576 7548 7297 7297
q2 2055 159 170 159
q3 10568 1172 1232 1172
q4 10228 820 829 820
q5 7747 3125 3069 3069
q6 236 148 146 146
q7 1028 610 612 610
q8 9356 1996 2063 1996
q9 6688 6518 6505 6505
q10 7070 2462 2406 2406
q11 452 251 257 251
q12 412 219 213 213
q13 17786 3027 3013 3013
q14 241 205 222 205
q15 569 545 516 516
q16 642 589 584 584
q17 988 569 582 569
q18 7534 6822 6617 6617
q19 1336 990 1085 990
q20 473 187 184 184
q21 4046 3388 3279 3279
q22 1126 1019 1027 1019
Total cold run time: 108157 ms
Total hot run time: 41620 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7280 7276 7282 7276
q2 322 224 227 224
q3 2952 2835 2793 2793
q4 1925 1747 1753 1747
q5 5491 5524 5558 5524
q6 224 136 139 136
q7 2081 1751 1708 1708
q8 3248 3426 3468 3426
q9 8603 8631 8634 8631
q10 3509 3493 3447 3447
q11 603 505 507 505
q12 787 571 591 571
q13 11688 3020 3027 3020
q14 288 264 256 256
q15 571 515 505 505
q16 685 638 624 624
q17 1847 1596 1602 1596
q18 7873 7556 7421 7421
q19 1675 1524 1546 1524
q20 2053 1834 1830 1830
q21 5452 5180 5345 5180
q22 1108 1048 1007 1007
Total cold run time: 70265 ms
Total hot run time: 58951 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2434695404
<details>
<summary>TPC-H: <b>Total hot run time: 41185 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccc3ae1721f8241c4724680b176424edc833a678, data reload: false
------ Round 1 ----------------------------------
q1 17571 7553 7303 7303
q2 2059 157 154 154
q3 10578 1135 1141 1135
q4 10244 851 880 851
q5 7762 3077 3048 3048
q6 241 152 154 152
q7 1007 603 593 593
q8 9363 1850 1968 1850
q9 6656 6442 6438 6438
q10 7019 2393 2410 2393
q11 454 242 249 242
q12 414 217 221 217
q13 17782 2988 2999 2988
q14 249 211 223 211
q15 563 533 514 514
q16 639 589 603 589
q17 965 533 634 533
q18 7492 6665 6835 6665
q19 1364 932 990 932
q20 499 184 184 184
q21 4031 3253 3172 3172
q22 1098 1025 1021 1021
Total cold run time: 108050 ms
Total hot run time: 41185 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7312 7262 7265 7262
q2 325 236 226 226
q3 2859 2782 2787 2782
q4 1951 1758 1643 1643
q5 5413 5450 5425 5425
q6 227 142 142 142
q7 2106 1736 1697 1697
q8 3232 3409 3425 3409
q9 8558 8542 8571 8542
q10 3480 3448 3418 3418
q11 579 477 483 477
q12 843 587 589 587
q13 7556 3049 3022 3022
q14 295 271 265 265
q15 570 512 515 512
q16 696 645 630 630
q17 1784 1587 1585 1585
q18 7891 7349 7590 7349
q19 1694 1440 1466 1440
q20 2045 1815 1829 1815
q21 5404 5282 5248 5248
q22 1115 1042 1016 1016
Total cold run time: 65935 ms
Total hot run time: 58492 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2527465055
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2527637217
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2527711725
<details>
<summary>TPC-H: <b>Total hot run time: 39880 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9cb6827dd47fde8bd54e1ecd5dd2aa86454a5a66, data reload: false
------ Round 1 ----------------------------------
q1 17589 7436 7277 7277
q2 2035 178 165 165
q3 10612 1094 1173 1094
q4 10547 729 745 729
q5 7617 2679 2651 2651
q6 245 147 143 143
q7 1018 627 605 605
q8 9248 1885 1916 1885
q9 6581 6413 6446 6413
q10 7021 2289 2299 2289
q11 465 265 261 261
q12 421 223 231 223
q13 17790 3024 3081 3024
q14 232 220 215 215
q15 583 532 521 521
q16 675 615 600 600
q17 980 518 572 518
q18 7122 6728 6637 6637
q19 1334 938 976 938
q20 460 185 181 181
q21 4006 3196 3220 3196
q22 375 320 315 315
Total cold run time: 106956 ms
Total hot run time: 39880 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7247 7228 7263 7228
q2 325 233 232 232
q3 2932 2818 2969 2818
q4 2097 1847 1824 1824
q5 5629 5640 5660 5640
q6 223 145 141 141
q7 2269 1790 1803 1790
q8 3384 3566 3482 3482
q9 8973 9030 8959 8959
q10 3588 3558 3566 3558
q11 604 509 519 509
q12 800 607 618 607
q13 12028 3215 3226 3215
q14 306 272 289 272
q15 564 543 549 543
q16 699 652 640 640
q17 1868 1651 1630 1630
q18 8294 7725 7373 7373
q19 1689 1570 1556 1556
q20 2105 1906 1919 1906
q21 5559 5485 5544 5485
q22 674 569 569 569
Total cold run time: 71857 ms
Total hot run time: 59977 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #42399:
URL: https://github.com/apache/doris/pull/42399#issuecomment-2530224423
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics)Support auto analyze columns that haven't been analyzed for a long time. [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #42399:
URL: https://github.com/apache/doris/pull/42399#discussion_r1853349796
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/LongTimeJobAppender.java:
##########
@@ -0,0 +1,160 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.statistics;
+
+import org.apache.doris.catalog.DatabaseIf;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.TableIf;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.Pair;
+import org.apache.doris.common.util.MasterDaemon;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.hive.HMSExternalTable;
+import org.apache.doris.statistics.util.StatisticsUtil;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
+
+/**
+ * This is a daemon thread to append long time not analyzed columns to long time job queue in AnalysisManager.
+ * Using this separate thread instead of using the existing StatisticsJobAppender is to avoid blocking other
+ * higher priority jobs.
+ */
+public class LongTimeJobAppender extends MasterDaemon {
Review Comment:
```suggestion
public class LongTimeStatisticJobAppender extends MasterDaemon {
```
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/ShowAutoAnalyzeJobsStmt.java:
##########
@@ -40,7 +40,7 @@
* [TABLE]
* [
* WHERE
- * [PRIORITY = ["HIGH"|"MID"|"LOW"]]
+ * [PRIORITY = ["HIGH"|"MID"|"LOW"|"LONG_TIME"]]
Review Comment:
How about "VERY_LOW"?
LONG_TIME is not a "priority"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org