You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2026/10/17 00:37:30 UTC
[PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
englefly opened a new pull request, #69209:
URL: https://github.com/apache/doris/pull/69209
### What problem does this PR solve?
Issue Number: None
Related PR: #52048
Problem Summary:
`LINEAR_ANALYZE_TEMPLATE` collected hot values in a third CTE: it grouped the sampled column
by value, counted each group, and kept the top `hot_value_collect_count` groups. That CTE
re-reads the whole sample, and it is the dominant cost of the template - the one sample
analyze picks for a single distribution column / single unique key column. The same template
also serves the full-scan fallback of a sample analyze (the requested sample rows exceed the
table, or the picked tablets hold fewer rows than requested), where the extra aggregation is
not bounded by the sample size at all.
This PR removes that CTE. `hot_value` keeps its position in the result set and is reported as
NULL, exactly like `FULL_ANALYZE_WITHOUT_HOT_VALUE_TEMPLATE` and `MERGE_PARTITION_TEMPLATE`
already do, and every reader already treats NULL as "not collected"
(`ColumnStatistic.hotValues == null`). Hot values are still collected by
`DUJ1_ANALYZE_TEMPLATE`, which derives them from the grouped sample it already builds for the
NDV estimate, and by a full analyze that asks for them explicitly
(`ANALYZE TABLE ... WITH HOT VALUE`).
The params that only served the removed CTE (`subStringColName`, `rowCount2` on the linear
path, `hotValueCollectCount`) are now filled for the DUJ1 algorithm only. As a side effect,
the linear branch of `PluginDrivenSampleAnalysisTask` no longer renders an unsubstituted
`${subStringColName}` into its SQL: it never set that param (the legacy `HMSAnalysisTask` it
was ported from did not either), so a bucketed plain-hive table that took that branch got
invalid SQL before this change.
Templates checked, and why they are left alone:
| Template | Hot value | Action |
|---|---|---|
| `LINEAR_ANALYZE_TEMPLATE` | collected via an extra GROUP BY over the sample | **removed here** |
| `DUJ1_ANALYZE_TEMPLATE` | collected from the grouped sample it already builds for NDV | kept |
| `FULL_ANALYZE_TEMPLATE` | only when the user writes `WITH HOT VALUE` | kept (explicit opt-in, not a default cost) |
| `FULL_ANALYZE_WITHOUT_HOT_VALUE_TEMPLATE`, `MERGE_PARTITION_TEMPLATE` | `null as hot_value` | unchanged |
| `PARTITION_ANALYZE_TEMPLATE`, `ANALYZE_PARTITION_COLUMN_TEMPLATE` | no hot value | unchanged |
Behaviour changes:
- A sample analyze no longer collects hot values for columns estimated with the linear
estimator: a single distribution column / single unique key column, and any sample analyze
that falls back to a full table scan.
- `ANALYZE ... WITH SAMPLE ... WITH HOT VALUE` was already rejected before this change; its
message changes from "Sample analyze always collects hot value" to "Sample analyze does not
support WITH HOT VALUE", because the old wording is no longer true.
### Release note
Sample analyze no longer collects hot values for columns that use the linear NDV estimator
(single distribution / single key column, and the full-scan fallback of a sample analyze),
which removes a second aggregation over the sampled data. Hot values are still collected by
sample analyze for columns estimated with DUJ1, and by `ANALYZE TABLE ... WITH HOT VALUE` in a
full analyze.
### Check List (For Author)
- Test
- [x] Unit Test
- `OlapAnalysisTaskTest` (25 tests, includes the linear-vs-DUJ1 hot value split)
- `AnalyzeTableCommandTest` (5 tests, includes the `WITH HOT VALUE` rejection message)
- [x] Regression test
- `test_hot_value` and `test_full_analyze_hot_value` updated for the new behaviour;
they need a built FE + cluster, so they run in CI
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [ ] No.
- [x] Yes. Sample analyze no longer collects hot values for columns analyzed by the linear
estimator, and the `WITH HOT VALUE` rejection message for sample analyze changed.
- Does this need documentation?
- [ ] No.
- [x] Yes. Any statement that sample analyze collects hot values needs to be narrowed; no
docs PR yet.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6228353159
<details>
<summary>ClickBench: <b>Total hot run time: 24.43 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3805adec90b637ce78a15c69745dd730e7555712, data reload: false
query1 0.00 0.00 0.00
query2 0.09 0.05 0.05
query3 0.26 0.12 0.11
query4 1.61 0.12 0.12
query5 0.27 0.24 0.24
query6 1.17 0.68 0.69
query7 0.03 0.01 0.00
query8 0.05 0.04 0.04
query9 0.40 0.35 0.35
query10 0.61 0.58 0.61
query11 0.20 0.14 0.14
query12 0.18 0.15 0.14
query13 0.51 0.49 0.52
query14 1.02 1.00 1.01
query15 0.65 0.62 0.61
query16 0.31 0.34 0.35
query17 1.13 1.14 1.09
query18 0.22 0.20 0.22
query19 2.12 1.94 1.94
query20 0.02 0.01 0.02
query21 15.51 0.27 0.14
query22 4.51 0.06 0.05
query23 16.27 0.31 0.12
query24 2.86 0.65 0.39
query25 0.11 0.07 0.04
query26 1.37 0.24 0.15
query27 0.04 0.03 0.05
query28 2.05 0.62 0.39
query29 12.50 4.20 3.31
query30 0.27 0.14 0.14
query31 2.78 0.58 0.34
query32 3.22 0.61 0.52
query33 3.16 3.19 3.17
query34 15.45 4.02 3.42
query35 3.32 3.33 3.35
query36 0.56 0.44 0.41
query37 0.10 0.06 0.06
query38 0.05 0.04 0.03
query39 0.03 0.03 0.03
query40 0.16 0.15 0.15
query41 0.08 0.03 0.02
query42 0.03 0.02 0.02
query43 0.05 0.04 0.04
Total cold run time: 95.33 s
Total hot run time: 24.43 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6228850902
# FE UT Coverage Report
Increment line coverage ` 85.71% (6/7)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69209_3805adec90b637ce78a15c69745dd730e7555712/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69209_3805adec90b637ce78a15c69745dd730e7555712/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6227432307
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6228309143
<details>
<summary>TPC-DS: <b>Total hot run time: 151957 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3805adec90b637ce78a15c69745dd730e7555712, data reload: false
query5 4296 574 448 448
query6 452 200 187 187
query7 4806 440 217 217
query8 318 175 154 154
query9 8767 3976 3933 3933
query10 467 300 275 275
query11 6045 3829 3534 3534
query12 146 93 88 88
query13 1273 440 316 316
query14 6081 4033 3777 3777
query14_1 3552 3653 3521 3521
query15 206 202 180 180
query16 943 453 407 407
query17 865 655 557 557
query18 2413 425 303 303
query19 201 169 142 142
query20 84 83 84 83
query21 212 135 110 110
query22 12921 13003 12972 12972
query23 13494 12756 12415 12415
query23_1 12567 12639 12565 12565
query24 7096 832 467 467
query24_1 501 465 484 465
query25 534 415 367 367
query26 1273 255 175 175
query27 2739 430 263 263
query28 4629 1946 1929 1929
query29 1540 580 435 435
query30 305 216 183 183
query31 984 915 793 793
query32 143 94 87 87
query33 507 295 235 235
query34 892 853 518 518
query35 740 753 638 638
query36 837 849 740 740
query37 125 99 91 91
query38 1771 1782 1657 1657
query39 796 812 753 753
query39_1 749 730 744 730
query40 220 113 94 94
query41 67 62 62 62
query42 86 83 81 81
query43 364 382 319 319
query44 1284 673 683 673
query45 178 174 163 163
query46 832 947 563 563
query47 3041 2987 2895 2895
query48 298 306 215 215
query49 649 414 303 303
query50 666 269 226 226
query51 10090 10336 10651 10336
query52 78 85 71 71
query53 178 209 151 151
query54 243 189 171 171
query55 71 69 65 65
query56 229 233 227 227
query57 1602 1578 1559 1559
query58 285 261 263 261
query59 2168 2148 2004 2004
query60 317 239 221 221
query61 148 145 148 145
query62 384 335 280 280
query63 198 165 160 160
query64 2720 981 794 794
query65 3278 3229 3222 3222
query66 1765 409 296 296
query67 20040 20334 20242 20242
query68 2963 988 605 605
query69 393 287 239 239
query70 853 816 786 786
query71 291 223 215 215
query72 2624 2594 2132 2132
query73 518 527 286 286
query74 4585 4455 4282 4282
query75 2327 2260 1937 1937
query76 2304 1066 595 595
query77 336 375 281 281
query78 9672 9738 9104 9104
query79 884 838 507 507
query80 470 414 325 325
query81 513 323 277 277
query82 215 132 108 108
query83 290 197 175 175
query84 302 119 107 107
query85 848 428 371 371
query86 258 238 221 221
query87 1998 1975 1812 1812
query88 3543 2686 2666 2666
query89 330 291 249 249
query90 1948 173 177 173
query91 156 152 122 122
query92 93 97 89 89
query93 975 991 567 567
query94 444 320 259 259
query95 588 342 382 342
query96 664 516 228 228
query97 2465 2401 2301 2301
query98 157 151 147 147
query99 776 759 652 652
Total cold run time: 232305 ms
Total hot run time: 151957 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6229645411
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6229645025
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6227955439
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "RoanHeNaN (via GitHub) via commits" <co...@doris.apache.org>.
RoanHeNaN commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6567237231
Can we pick this pr to branch 4.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6228215097
<details>
<summary>TPC-H: <b>Total hot run time: 29112 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3805adec90b637ce78a15c69745dd730e7555712, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17577 5588 5504 5504
q2 2075 310 246 246
q3 10285 1385 829 829
q4 4671 480 343 343
q5 7450 791 518 518
q6 195 189 154 154
q7 745 799 606 606
q8 9253 1302 1082 1082
q9 5547 4240 4249 4240
q10 6790 1309 1036 1036
q11 440 272 235 235
q12 639 400 280 280
q13 18073 3487 2955 2955
q14 276 275 248 248
q15 q16 731 725 679 679
q17 1188 751 577 577
q18 7010 6233 6188 6188
q19 1098 962 614 614
q20 391 340 241 241
q21 5021 2271 2401 2271
q22 366 323 266 266
Total cold run time: 99821 ms
Total hot run time: 29112 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6117 6100 6116 6100
q2 671 588 545 545
q3 5216 5365 4909 4909
q4 2011 2102 1537 1537
q5 5245 5169 6016 5169
q6 261 198 160 160
q7 2267 2030 1841 1841
q8 3124 2707 2648 2648
q9 8103 7981 8052 7981
q10 4003 3705 3359 3359
q11 647 451 417 417
q12 700 704 533 533
q13 3592 3879 3277 3277
q14 301 316 289 289
q15 q16 706 717 656 656
q17 8247 6905 6861 6861
q18 13152 12293 13003 12293
q19 904 773 776 773
q20 2199 2165 1935 1935
q21 5753 4677 4730 4677
q22 496 474 415 415
Total cold run time: 73715 ms
Total hot run time: 66375 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #69209:
URL: https://github.com/apache/doris/pull/69209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Stop collecting hot value in the linear sample estimator [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69209:
URL: https://github.com/apache/doris/pull/69209#issuecomment-6227955668
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org