You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2026/04/09 11:55:14 UTC
[PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
yujun777 opened a new pull request, #62295:
URL: https://github.com/apache/doris/pull/62295
[improvement](statistics) Collect hot value during full statistics analysis
### What problem does this PR solve?
Issue Number: close #xxx
Problem Summary: Previously, full statistics collection (`ANALYZE TABLE ... WITH SYNC`)
hardcoded `null as hot_value` in `FULL_ANALYZE_TEMPLATE`, meaning hot values (frequent
values) were only collected during sample-based analysis. This made full analysis
produce less complete statistics than sample analysis.
This change rewrites `FULL_ANALYZE_TEMPLATE` to use a CTE-based structure (matching
`LINEAR_ANALYZE_TEMPLATE` and `DUJ1_ANALYZE_TEMPLATE` patterns) that computes hot values
via GROUP BY + TOP-N aggregation. Both `OlapAnalysisTask` and `ExternalAnalysisTask` are
updated to pass the required parameters (`hotValueCollectCount`, `subStringColName`,
`rowCount2`) to the template.
### Release note
Full statistics collection now also collects hot value (frequent value) information,
matching the behavior of sample-based collection.
### Check List (For Author)
- Test: Regression test (test_hot_value, test_full_analyze_hot_value) / Unit Test (OlapAnalysisTaskTest, HMSAnalysisTaskTest)
- Behavior changed: Yes - full analyze now produces hot_value instead of null
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #62295:
URL: https://github.com/apache/doris/pull/62295#discussion_r3057704896
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java:
##########
@@ -328,7 +328,11 @@ protected void doFull() throws Exception {
if (StatisticsUtil.enablePartitionAnalyze() && tbl.isPartitionedTable()) {
doPartitionTable();
} else {
- StringSubstitutor stringSubstitutor = new StringSubstitutor(buildSqlParams());
+ Map<String, String> params = buildSqlParams();
Review Comment:
we no handle with case enable_partition_analyze = true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #62295:
URL: https://github.com/apache/doris/pull/62295#issuecomment-4214006542
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #62295:
URL: https://github.com/apache/doris/pull/62295#issuecomment-4214006643
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62295:
URL: https://github.com/apache/doris/pull/62295#issuecomment-4233550214
# FE UT Coverage Report
Increment line coverage ` 100.00% (8/8)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62295_4b9481b1a519191b6d1535bf219c49bc164538c9/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62295_4b9481b1a519191b6d1535bf219c49bc164538c9/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #62295:
URL: https://github.com/apache/doris/pull/62295#issuecomment-4233319698
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #62295:
URL: https://github.com/apache/doris/pull/62295#issuecomment-4215210872
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 closed pull request #62295: [improvement](statistics) Collect hot value during full statistics analysis
URL: https://github.com/apache/doris/pull/62295
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #62295:
URL: https://github.com/apache/doris/pull/62295#issuecomment-4214006054
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Collect hot value during full statistics analysis [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #62295:
URL: https://github.com/apache/doris/pull/62295#discussion_r3057687174
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java:
##########
@@ -328,7 +328,11 @@ protected void doFull() throws Exception {
if (StatisticsUtil.enablePartitionAnalyze() && tbl.isPartitionedTable()) {
doPartitionTable();
} else {
- StringSubstitutor stringSubstitutor = new StringSubstitutor(buildSqlParams());
+ Map<String, String> params = buildSqlParams();
Review Comment:
This only fixes the non-partition full-analyze branch. When `enable_partition_analyze` is true and the table is partitioned, `doFull()` still goes through `doPartitionTable()`, which eventually merges with `MERGE_PARTITION_TEMPLATE` in `BaseAnalysisTask` and that template still hardcodes `null as hot_value`. In that configuration, `ANALYZE TABLE ... WITH SYNC` will continue to produce no hot values for partitioned OLAP tables, so the behavior change described in the PR is still incomplete. The new tests all stub `enablePartitionAnalyze()` to `false`, so they won't catch this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org