You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "freemandealer (via GitHub)" <gi...@apache.org> on 2025/04/24 11:03:02 UTC

[PR] [fix](cloud) limit amount of ttl cache blocks gc once in a time [doris]

freemandealer opened a new pull request, #50387:
URL: https://github.com/apache/doris/pull/50387

   too many ttl cache blocks gc will burst the cache lock latency and thus affect the query latency. limit them into batches to unleash the lock.
   
   ### What problem does this PR solve?
   
   Issue Number: close #xxx
   
   Related PR: #xxx
   
   Problem Summary:
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [ ] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [ ] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [ ] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) limit amount of ttl cache blocks gc once in a time [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #50387:
URL: https://github.com/apache/doris/pull/50387#issuecomment-2827229973

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) batch process ttl cache block gc to limit lock held time once in a time [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50387:
URL: https://github.com/apache/doris/pull/50387#issuecomment-2827389032

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) batch process ttl cache block gc to limit lock held time once in a time [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50387:
URL: https://github.com/apache/doris/pull/50387#issuecomment-2827893144

   # BE UT Coverage Report
   Increment line coverage `100.00% (10/10)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/1b3cfeeefe009b77e10656104eec6f3853f7f3e7_1b3cfeeefe009b77e10656104eec6f3853f7f3e7/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/1b3cfeeefe009b77e10656104eec6f3853f7f3e7_1b3cfeeefe009b77e10656104eec6f3853f7f3e7/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 53.75% (14595/27151) |
   | Line Coverage     | 42.58% (126691/297530) |
   | Region Coverage   | 41.38% (64768/156505) |
   | Branch Coverage   | 35.93% (32558/90608) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) batch process ttl cache block gc to limit lock held time once in a time [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50387:
URL: https://github.com/apache/doris/pull/50387#issuecomment-2827388962

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) batch process ttl cache block gc to limit lock held time once in a time [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50387:
URL: https://github.com/apache/doris/pull/50387#issuecomment-2827553719

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 29.84 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 1b3cfeeefe009b77e10656104eec6f3853f7f3e7, data reload: false
   
   query1	0.04	0.04	0.03
   query2	0.13	0.11	0.11
   query3	0.26	0.21	0.19
   query4	1.88	0.20	0.20
   query5	0.60	0.59	0.60
   query6	1.32	0.73	0.72
   query7	0.02	0.02	0.01
   query8	0.05	0.04	0.03
   query9	0.60	0.52	0.52
   query10	0.57	0.57	0.57
   query11	0.16	0.11	0.12
   query12	0.15	0.11	0.11
   query13	0.60	0.60	0.59
   query14	1.16	1.21	1.20
   query15	0.87	0.83	0.84
   query16	0.38	0.38	0.38
   query17	1.08	1.01	1.04
   query18	0.21	0.20	0.20
   query19	1.89	1.78	1.78
   query20	0.02	0.01	0.01
   query21	15.41	0.93	0.55
   query22	0.74	1.19	0.69
   query23	14.87	1.41	0.60
   query24	7.50	0.97	1.32
   query25	0.44	0.26	0.09
   query26	0.62	0.16	0.14
   query27	0.05	0.05	0.04
   query28	9.67	0.82	0.45
   query29	12.59	3.94	3.30
   query30	0.25	0.09	0.06
   query31	2.81	0.58	0.37
   query32	3.24	0.56	0.47
   query33	3.15	3.05	3.11
   query34	15.71	5.08	4.52
   query35	4.52	4.57	4.54
   query36	0.65	0.49	0.49
   query37	0.09	0.06	0.06
   query38	0.05	0.04	0.03
   query39	0.03	0.03	0.03
   query40	0.17	0.15	0.13
   query41	0.09	0.03	0.02
   query42	0.03	0.03	0.02
   query43	0.04	0.04	0.03
   Total cold run time: 104.71 s
   Total hot run time: 29.84 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) limit amount of ttl cache blocks gc once in a time [doris]

Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #50387:
URL: https://github.com/apache/doris/pull/50387#issuecomment-2827209942

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) batch process ttl cache block gc to limit lock held time once in a time [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50387:
URL: https://github.com/apache/doris/pull/50387#issuecomment-2828576868

   # BE Regression P0 && UT Coverage Report
   Increment line coverage `20.00% (2/10)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/50387_1b3cfeeefe009b77e10656104eec6f3853f7f3e7_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/50387_1b3cfeeefe009b77e10656104eec6f3853f7f3e7_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage |                    |
   | Line Coverage     |                    |
   | Region Coverage   |                    |
   | Branch Coverage   |                    |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) batch process ttl cache block gc to limit lock held time once in a time [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50387:
URL: https://github.com/apache/doris/pull/50387#issuecomment-2827496270

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 34231 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 1b3cfeeefe009b77e10656104eec6f3853f7f3e7, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	26445	5104	5038	5038
   q2	2084	275	188	188
   q3	10499	1248	703	703
   q4	10239	1017	545	545
   q5	7667	2441	2377	2377
   q6	193	168	136	136
   q7	938	741	626	626
   q8	9340	1344	1195	1195
   q9	6883	5130	5173	5130
   q10	6824	2312	1890	1890
   q11	488	280	277	277
   q12	347	353	211	211
   q13	17789	3736	3049	3049
   q14	230	225	214	214
   q15	542	480	486	480
   q16	452	444	403	403
   q17	604	888	374	374
   q18	7623	7239	7219	7219
   q19	1364	954	571	571
   q20	336	336	220	220
   q21	3908	3386	2426	2426
   q22	1074	1038	959	959
   Total cold run time: 115869 ms
   Total hot run time: 34231 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	5191	5165	5109	5109
   q2	241	328	238	238
   q3	2148	2657	2306	2306
   q4	1440	1826	1514	1514
   q5	4577	4494	4413	4413
   q6	250	167	128	128
   q7	2036	1923	1741	1741
   q8	2597	2629	2513	2513
   q9	7308	7263	7175	7175
   q10	3017	3216	2755	2755
   q11	577	510	493	493
   q12	709	762	599	599
   q13	3638	4048	3318	3318
   q14	316	309	281	281
   q15	521	491	472	472
   q16	449	495	462	462
   q17	1169	1600	1341	1341
   q18	7915	7499	7519	7499
   q19	802	831	810	810
   q20	1996	1954	1797	1797
   q21	5206	4782	4667	4667
   q22	1053	1044	1004	1004
   Total cold run time: 53156 ms
   Total hot run time: 50635 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) batch process ttl cache block gc to limit lock held time once in a time [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50387:
URL: https://github.com/apache/doris/pull/50387#issuecomment-2827537031

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 185546 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 1b3cfeeefe009b77e10656104eec6f3853f7f3e7, data reload: false
   
   query1	1025	493	496	493
   query2	6572	1886	1804	1804
   query3	6747	222	214	214
   query4	26463	23434	23189	23189
   query5	4324	614	486	486
   query6	298	193	179	179
   query7	4610	476	287	287
   query8	293	243	231	231
   query9	8619	2575	2575	2575
   query10	447	317	296	296
   query11	15265	15247	14861	14861
   query12	177	109	109	109
   query13	1645	514	400	400
   query14	8773	6233	6157	6157
   query15	214	188	167	167
   query16	7149	616	451	451
   query17	1140	781	550	550
   query18	1970	392	291	291
   query19	182	178	153	153
   query20	120	121	116	116
   query21	214	121	103	103
   query22	4122	4178	4060	4060
   query23	34027	32952	32833	32833
   query24	8454	2354	2340	2340
   query25	542	455	380	380
   query26	1227	263	150	150
   query27	2774	497	326	326
   query28	4326	2120	2096	2096
   query29	738	558	422	422
   query30	279	216	196	196
   query31	939	875	762	762
   query32	73	71	65	65
   query33	571	375	316	316
   query34	804	834	514	514
   query35	770	822	728	728
   query36	973	983	898	898
   query37	119	111	78	78
   query38	4264	4097	4039	4039
   query39	1440	1408	1407	1407
   query40	216	115	109	109
   query41	57	60	53	53
   query42	120	104	111	104
   query43	481	507	480	480
   query44	1276	788	779	779
   query45	176	170	171	170
   query46	832	1020	614	614
   query47	1795	1825	1760	1760
   query48	365	404	301	301
   query49	779	532	421	421
   query50	639	656	401	401
   query51	4081	4154	4028	4028
   query52	107	101	92	92
   query53	214	252	184	184
   query54	581	571	504	504
   query55	90	85	92	85
   query56	312	293	303	293
   query57	1160	1169	1081	1081
   query58	263	264	242	242
   query59	2643	2668	2617	2617
   query60	320	317	320	317
   query61	130	125	127	125
   query62	813	723	701	701
   query63	223	186	178	178
   query64	4351	987	663	663
   query65	4304	4284	4320	4284
   query66	1139	403	308	308
   query67	16003	15397	15511	15397
   query68	7813	864	502	502
   query69	469	289	262	262
   query70	1177	1174	1081	1081
   query71	417	309	289	289
   query72	5565	4782	4981	4782
   query73	709	661	339	339
   query74	8989	8890	8944	8890
   query75	3389	3242	2749	2749
   query76	3342	1186	750	750
   query77	666	459	280	280
   query78	10096	10379	9269	9269
   query79	2449	821	573	573
   query80	645	526	424	424
   query81	494	259	216	216
   query82	220	125	99	99
   query83	253	247	236	236
   query84	278	102	82	82
   query85	760	425	308	308
   query86	373	313	296	296
   query87	4328	4576	4231	4231
   query88	3790	2189	2175	2175
   query89	374	316	284	284
   query90	2023	219	208	208
   query91	141	141	121	121
   query92	79	63	59	59
   query93	2248	935	564	564
   query94	655	415	301	301
   query95	368	285	290	285
   query96	485	571	273	273
   query97	3076	3200	3126	3126
   query98	226	210	207	207
   query99	1355	1417	1272	1272
   Total cold run time: 273411 ms
   Total hot run time: 185546 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](cloud) batch process ttl cache block gc to limit lock held time once in a time [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #50387:
URL: https://github.com/apache/doris/pull/50387


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org