You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2026/10/18 02:02:24 UTC

[PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

englefly opened a new pull request, #69225:
URL: https://github.com/apache/doris/pull/69225

   ### What problem does this PR solve?
   
   Issue Number: None
   
   Related PR: #69209
   
   Problem Summary:
   
   `ANALYZE ... WITH SAMPLE ... WITH HOT VALUE` was rejected by `AnalyzeCommand.validate`. The
   rejection was introduced together with `WITH HOT VALUE`: at that time a sample analyze always
   collected hot values, so the flag was redundant and rejecting it early kept the command from
   silently doing nothing.
   
   After #69209 the linear estimator no longer collects hot values, but a sample analyze still
   collects them for every column the DUJ1 estimator serves - and which estimator a column gets is
   decided per column at execution time (`OlapAnalysisTask.getSampleCollectInfo`: table row count,
   sampled rows, single distribution / single key column, partition flags), so the command layer
   cannot tell whether the flag would be honored for a particular column. Rejecting the combination
   is therefore wrong for the columns that do collect hot values (the DUJ1 ones), while for the
   linear ones the flag never could force collection anyway.
   
   This PR accepts the combination. The flag keeps its only real meaning - it selects the hot value
   template of a full analyze - and has no effect on a sample analyze: hot values still come from the
   DUJ1 estimator, and a full analyze with `WITH HOT VALUE` remains the way to collect them for a
   linear-estimated column.
   
   Before this PR, both command paths that reach the check (`ANALYZE TABLE` and `ANALYZE DATABASE`)
   returned:
   
   ```
   ERROR 1105 (HY000): errCode = 2, detailMessage = Sample analyze does not support WITH HOT VALUE
   ```
   
   After it, the statement succeeds and the analyze runs with the estimator the sample picks.
   
   ### Release note
   
   `ANALYZE TABLE ... WITH SAMPLE ... WITH HOT VALUE` is no longer rejected. The flag only affects a
   full analyze; on a sample analyze hot values are still collected only for the columns analyzed by
   the DUJ1 estimator.
   
   ### Check List (For Author)
   
   - Test
       - [x] Unit Test
           - `AnalyzeTableCommandTest#testSampleAnalyzeWithHotValueAccepted` (the command validates
             without the hot-value rejection, and the parsed statement is still a sample analyze)
       - [x] Regression test
           - `test_full_analyze_hot_value`: Test 6 accepts the flag on a table whose sample falls back
             to the linear estimator (no hot value), Test 7 keeps the DUJ1 hot value with the flag
             present; needs a built FE + cluster, so it runs in CI
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
   - Behavior changed:
       - [ ] No.
       - [x] Yes. `ANALYZE ... WITH SAMPLE ... WITH HOT VALUE` no longer fails; the flag is accepted
         and has no effect on a sample analyze (hot values are decided by the estimator).
   - Does this need documentation?
       - [ ] No.
       - [x] Yes, ideally: the `ANALYZE` reference should say the flag is only meaningful for a full
         analyze. No docs PR yet.
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244685762

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244918823

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29005 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit a3e4f441aba6a75de252b5304431e8012c32beff, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17580	5650	5541	5541
   q2	2078	323	260	260
   q3	10241	1356	821	821
   q4	4683	464	337	337
   q5	7417	768	524	524
   q6	186	189	158	158
   q7	737	792	592	592
   q8	9235	1282	1068	1068
   q9	5465	4244	4231	4231
   q10	6806	1288	1045	1045
   q11	446	257	234	234
   q12	640	401	278	278
   q13	18064	3568	3010	3010
   q14	274	267	251	251
   q15	q16	715	723	673	673
   q17	1178	734	568	568
   q18	7058	6199	6149	6149
   q19	1100	946	580	580
   q20	386	334	227	227
   q21	4621	2194	2374	2194
   q22	356	307	264	264
   Total cold run time: 99266 ms
   Total hot run time: 29005 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	6182	6107	6089	6089
   q2	671	578	542	542
   q3	5232	5364	4987	4987
   q4	1987	2101	1521	1521
   q5	5276	5123	5802	5123
   q6	263	209	160	160
   q7	2309	2085	1865	1865
   q8	3145	2767	2745	2745
   q9	8137	8109	8214	8109
   q10	3779	3753	3328	3328
   q11	677	446	423	423
   q12	697	706	511	511
   q13	3530	3845	3229	3229
   q14	318	306	294	294
   q15	q16	696	735	672	672
   q17	8314	7029	6818	6818
   q18	13132	12226	13071	12226
   q19	905	815	797	797
   q20	2183	2188	1942	1942
   q21	5703	4671	4766	4671
   q22	495	463	419	419
   Total cold run time: 73631 ms
   Total hot run time: 66471 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244992590

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 151779 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit a3e4f441aba6a75de252b5304431e8012c32beff, data reload: false
   
   query5	4306	585	449	449
   query6	420	207	191	191
   query7	4803	436	232	232
   query8	323	165	155	155
   query9	8739	3945	3951	3945
   query10	463	303	257	257
   query11	6021	3794	3532	3532
   query12	148	99	86	86
   query13	1257	438	316	316
   query14	6127	4063	3706	3706
   query14_1	3491	3528	3549	3528
   query15	207	200	185	185
   query16	942	438	347	347
   query17	867	662	533	533
   query18	2442	426	330	330
   query19	203	170	135	135
   query20	85	87	85	85
   query21	214	137	109	109
   query22	12954	12966	12830	12830
   query23	13371	12795	12521	12521
   query23_1	12563	12562	12612	12562
   query24	7117	835	466	466
   query24_1	474	475	472	472
   query25	541	418	390	390
   query26	1331	265	151	151
   query27	2746	444	275	275
   query28	4603	1941	1945	1941
   query29	1583	608	489	489
   query30	298	221	186	186
   query31	1016	930	815	815
   query32	139	86	86	86
   query33	547	296	240	240
   query34	875	846	507	507
   query35	760	754	644	644
   query36	790	806	719	719
   query37	130	99	83	83
   query38	1774	1734	1665	1665
   query39	783	800	744	744
   query39_1	734	721	739	721
   query40	219	110	93	93
   query41	65	66	61	61
   query42	86	80	83	80
   query43	364	378	325	325
   query44	1294	692	692	692
   query45	186	188	162	162
   query46	871	946	569	569
   query47	3025	3003	2934	2934
   query48	293	300	222	222
   query49	576	403	295	295
   query50	677	271	220	220
   query51	10460	10218	10295	10218
   query52	78	81	70	70
   query53	182	203	149	149
   query54	237	208	188	188
   query55	75	67	64	64
   query56	233	212	232	212
   query57	1631	1579	1569	1569
   query58	286	271	254	254
   query59	2139	2059	1992	1992
   query60	280	241	226	226
   query61	155	148	152	148
   query62	396	338	281	281
   query63	195	164	162	162
   query64	2724	942	783	783
   query65	3310	3222	3247	3222
   query66	1765	413	299	299
   query67	19977	20169	19953	19953
   query68	3035	983	622	622
   query69	371	288	244	244
   query70	842	783	777	777
   query71	289	220	205	205
   query72	2889	2608	2214	2214
   query73	511	528	287	287
   query74	4580	4444	4282	4282
   query75	2345	2285	1959	1959
   query76	2300	1023	632	632
   query77	341	369	277	277
   query78	9741	9791	9143	9143
   query79	842	924	558	558
   query80	504	470	348	348
   query81	523	320	267	267
   query82	213	131	113	113
   query83	299	195	182	182
   query84	303	127	100	100
   query85	844	429	377	377
   query86	329	238	229	229
   query87	2013	1983	1805	1805
   query88	3583	2687	2673	2673
   query89	325	294	259	259
   query90	1806	177	179	177
   query91	158	149	125	125
   query92	98	88	89	88
   query93	985	992	572	572
   query94	453	293	270	270
   query95	573	408	324	324
   query96	658	526	236	236
   query97	2475	2397	2311	2311
   query98	158	153	146	146
   query99	780	759	642	642
   Total cold run time: 232952 ms
   Total hot run time: 151779 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244520411

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6245030967

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 24.39 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit a3e4f441aba6a75de252b5304431e8012c32beff, data reload: false
   
   query1	0.01	0.00	0.00
   query2	0.10	0.05	0.05
   query3	0.27	0.12	0.12
   query4	1.61	0.12	0.12
   query5	0.24	0.24	0.25
   query6	1.16	0.68	0.68
   query7	0.04	0.01	0.01
   query8	0.06	0.04	0.04
   query9	0.40	0.33	0.32
   query10	0.57	0.58	0.59
   query11	0.21	0.16	0.14
   query12	0.19	0.15	0.14
   query13	0.49	0.48	0.49
   query14	1.04	1.01	1.01
   query15	0.62	0.60	0.62
   query16	0.34	0.33	0.31
   query17	1.14	1.16	1.14
   query18	0.22	0.21	0.20
   query19	2.11	2.05	1.97
   query20	0.02	0.02	0.01
   query21	15.44	0.26	0.14
   query22	4.52	0.05	0.04
   query23	16.26	0.32	0.12
   query24	3.28	0.61	0.38
   query25	0.10	0.05	0.06
   query26	1.03	0.24	0.15
   query27	0.05	0.03	0.04
   query28	1.93	0.59	0.39
   query29	12.53	4.21	3.32
   query30	0.28	0.14	0.13
   query31	2.78	0.57	0.33
   query32	3.23	0.60	0.52
   query33	3.13	3.25	3.10
   query34	15.56	3.98	3.38
   query35	3.34	3.36	3.35
   query36	0.55	0.45	0.47
   query37	0.09	0.07	0.06
   query38	0.06	0.04	0.04
   query39	0.04	0.03	0.03
   query40	0.17	0.15	0.14
   query41	0.09	0.03	0.03
   query42	0.04	0.02	0.02
   query43	0.04	0.03	0.04
   Total cold run time: 95.38 s
   Total hot run time: 24.39 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6246282039

   # FE Regression Coverage Report
   Increment line coverage `  100% (0/0)` :tada:
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/69225_a3e4f441aba6a75de252b5304431e8012c32beff_merge_fe/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/69225_a3e4f441aba6a75de252b5304431e8012c32beff_merge_fe/report/index.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6284805088

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #69225:
URL: https://github.com/apache/doris/pull/69225


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6244685884

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](statistics) Accept WITH HOT VALUE on a sample analyze [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69225:
URL: https://github.com/apache/doris/pull/69225#issuecomment-6284805594

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org