You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2026/10/18 02:02:24 UTC
[PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
englefly opened a new pull request, #69225:
URL: https://github.com/apache/doris/pull/69225
### What problem does this PR solve?
Issue Number: None
Related PR: #69209
Problem Summary:
`ANALYZE ... WITH SAMPLE ... WITH HOT VALUE` was rejected by `AnalyzeCommand.validate`. The
rejection was introduced together with `WITH HOT VALUE`: at that time a sample analyze always
collected hot values, so the flag was redundant and rejecting it early kept the command from
silently doing nothing.
After #69209 the linear estimator no longer collects hot values, but a sample analyze still
collects them for every column the DUJ1 estimator serves - and which estimator a column gets is
decided per column at execution time (`OlapAnalysisTask.getSampleCollectInfo`: table row count,
sampled rows, single distribution / single key column, partition flags), so the command layer
cannot tell whether the flag would be honored for a particular column. Rejecting the combination
is therefore wrong for the columns that do collect hot values (the DUJ1 ones), while for the
linear ones the flag never could force collection anyway.
This PR accepts the combination. The flag keeps its only real meaning - it selects the hot value
template of a full analyze - and has no effect on a sample analyze: hot values still come from the
DUJ1 estimator, and a full analyze with `WITH HOT VALUE` remains the way to collect them for a
linear-estimated column.
Before this PR, both command paths that reach the check (`ANALYZE TABLE` and `ANALYZE DATABASE`)
returned:
```
ERROR 1105 (HY000): errCode = 2, detailMessage = Sample analyze does not support WITH HOT VALUE
```
After it, the statement succeeds and the analyze runs with the estimator the sample picks.
### Release note
`ANALYZE TABLE ... WITH SAMPLE ... WITH HOT VALUE` is no longer rejected. The flag only affects a
full analyze; on a sample analyze hot values are still collected only for the columns analyzed by
the DUJ1 estimator.
### Check List (For Author)
- Test
- [x] Unit Test
- `AnalyzeTableCommandTest#testSampleAnalyzeWithHotValueAccepted` (the command validates
without the hot-value rejection, and the parsed statement is still a sample analyze)
- [x] Regression test
- `test_full_analyze_hot_value`: Test 6 accepts the flag on a table whose sample falls back
to the linear estimator (no hot value), Test 7 keeps the DUJ1 hot value with the flag
present; needs a built FE + cluster, so it runs in CI
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. `ANALYZE ... WITH SAMPLE ... WITH HOT VALUE` no longer fails; the flag is accepted
and has no effect on a sample analyze (hot values are decided by the estimator).
- Does this need documentation?
- [ ] No.
- [x] Yes, ideally: the `ANALYZE` reference should say the flag is only meaningful for a full
analyze. No docs PR yet.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244685762
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244918823
<details>
<summary>TPC-H: <b>Total hot run time: 29005 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3e4f441aba6a75de252b5304431e8012c32beff, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17580 5650 5541 5541
q2 2078 323 260 260
q3 10241 1356 821 821
q4 4683 464 337 337
q5 7417 768 524 524
q6 186 189 158 158
q7 737 792 592 592
q8 9235 1282 1068 1068
q9 5465 4244 4231 4231
q10 6806 1288 1045 1045
q11 446 257 234 234
q12 640 401 278 278
q13 18064 3568 3010 3010
q14 274 267 251 251
q15 q16 715 723 673 673
q17 1178 734 568 568
q18 7058 6199 6149 6149
q19 1100 946 580 580
q20 386 334 227 227
q21 4621 2194 2374 2194
q22 356 307 264 264
Total cold run time: 99266 ms
Total hot run time: 29005 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6182 6107 6089 6089
q2 671 578 542 542
q3 5232 5364 4987 4987
q4 1987 2101 1521 1521
q5 5276 5123 5802 5123
q6 263 209 160 160
q7 2309 2085 1865 1865
q8 3145 2767 2745 2745
q9 8137 8109 8214 8109
q10 3779 3753 3328 3328
q11 677 446 423 423
q12 697 706 511 511
q13 3530 3845 3229 3229
q14 318 306 294 294
q15 q16 696 735 672 672
q17 8314 7029 6818 6818
q18 13132 12226 13071 12226
q19 905 815 797 797
q20 2183 2188 1942 1942
q21 5703 4671 4766 4671
q22 495 463 419 419
Total cold run time: 73631 ms
Total hot run time: 66471 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244992590
<details>
<summary>TPC-DS: <b>Total hot run time: 151779 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3e4f441aba6a75de252b5304431e8012c32beff, data reload: false
query5 4306 585 449 449
query6 420 207 191 191
query7 4803 436 232 232
query8 323 165 155 155
query9 8739 3945 3951 3945
query10 463 303 257 257
query11 6021 3794 3532 3532
query12 148 99 86 86
query13 1257 438 316 316
query14 6127 4063 3706 3706
query14_1 3491 3528 3549 3528
query15 207 200 185 185
query16 942 438 347 347
query17 867 662 533 533
query18 2442 426 330 330
query19 203 170 135 135
query20 85 87 85 85
query21 214 137 109 109
query22 12954 12966 12830 12830
query23 13371 12795 12521 12521
query23_1 12563 12562 12612 12562
query24 7117 835 466 466
query24_1 474 475 472 472
query25 541 418 390 390
query26 1331 265 151 151
query27 2746 444 275 275
query28 4603 1941 1945 1941
query29 1583 608 489 489
query30 298 221 186 186
query31 1016 930 815 815
query32 139 86 86 86
query33 547 296 240 240
query34 875 846 507 507
query35 760 754 644 644
query36 790 806 719 719
query37 130 99 83 83
query38 1774 1734 1665 1665
query39 783 800 744 744
query39_1 734 721 739 721
query40 219 110 93 93
query41 65 66 61 61
query42 86 80 83 80
query43 364 378 325 325
query44 1294 692 692 692
query45 186 188 162 162
query46 871 946 569 569
query47 3025 3003 2934 2934
query48 293 300 222 222
query49 576 403 295 295
query50 677 271 220 220
query51 10460 10218 10295 10218
query52 78 81 70 70
query53 182 203 149 149
query54 237 208 188 188
query55 75 67 64 64
query56 233 212 232 212
query57 1631 1579 1569 1569
query58 286 271 254 254
query59 2139 2059 1992 1992
query60 280 241 226 226
query61 155 148 152 148
query62 396 338 281 281
query63 195 164 162 162
query64 2724 942 783 783
query65 3310 3222 3247 3222
query66 1765 413 299 299
query67 19977 20169 19953 19953
query68 3035 983 622 622
query69 371 288 244 244
query70 842 783 777 777
query71 289 220 205 205
query72 2889 2608 2214 2214
query73 511 528 287 287
query74 4580 4444 4282 4282
query75 2345 2285 1959 1959
query76 2300 1023 632 632
query77 341 369 277 277
query78 9741 9791 9143 9143
query79 842 924 558 558
query80 504 470 348 348
query81 523 320 267 267
query82 213 131 113 113
query83 299 195 182 182
query84 303 127 100 100
query85 844 429 377 377
query86 329 238 229 229
query87 2013 1983 1805 1805
query88 3583 2687 2673 2673
query89 325 294 259 259
query90 1806 177 179 177
query91 158 149 125 125
query92 98 88 89 88
query93 985 992 572 572
query94 453 293 270 270
query95 573 408 324 324
query96 658 526 236 236
query97 2475 2397 2311 2311
query98 158 153 146 146
query99 780 759 642 642
Total cold run time: 232952 ms
Total hot run time: 151779 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244520411
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6245030967
<details>
<summary>ClickBench: <b>Total hot run time: 24.39 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3e4f441aba6a75de252b5304431e8012c32beff, data reload: false
query1 0.01 0.00 0.00
query2 0.10 0.05 0.05
query3 0.27 0.12 0.12
query4 1.61 0.12 0.12
query5 0.24 0.24 0.25
query6 1.16 0.68 0.68
query7 0.04 0.01 0.01
query8 0.06 0.04 0.04
query9 0.40 0.33 0.32
query10 0.57 0.58 0.59
query11 0.21 0.16 0.14
query12 0.19 0.15 0.14
query13 0.49 0.48 0.49
query14 1.04 1.01 1.01
query15 0.62 0.60 0.62
query16 0.34 0.33 0.31
query17 1.14 1.16 1.14
query18 0.22 0.21 0.20
query19 2.11 2.05 1.97
query20 0.02 0.02 0.01
query21 15.44 0.26 0.14
query22 4.52 0.05 0.04
query23 16.26 0.32 0.12
query24 3.28 0.61 0.38
query25 0.10 0.05 0.06
query26 1.03 0.24 0.15
query27 0.05 0.03 0.04
query28 1.93 0.59 0.39
query29 12.53 4.21 3.32
query30 0.28 0.14 0.13
query31 2.78 0.57 0.33
query32 3.23 0.60 0.52
query33 3.13 3.25 3.10
query34 15.56 3.98 3.38
query35 3.34 3.36 3.35
query36 0.55 0.45 0.47
query37 0.09 0.07 0.06
query38 0.06 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.14
query41 0.09 0.03 0.03
query42 0.04 0.02 0.02
query43 0.04 0.03 0.04
Total cold run time: 95.38 s
Total hot run time: 24.39 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6246282039
# FE Regression Coverage Report
Increment line coverage ` 100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69225_a3e4f441aba6a75de252b5304431e8012c32beff_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69225_a3e4f441aba6a75de252b5304431e8012c32beff_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6284805088
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #69225:
URL: https://github.com/apache/doris/pull/69225
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244685884
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6284805594
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org