You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "freemandealer (via GitHub)" <gi...@apache.org> on 2026/03/05 08:23:50 UTC

[PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

freemandealer opened a new pull request, #61072:
URL: https://github.com/apache/doris/pull/61072

   - decouple realtime FileCache profile updates from the local/remote-bytes branch and update once per cycle when file cache is enabled
   - reset file_cache_stats as a whole (`= {}`) after realtime reporting, instead of resetting only bytes_read_from_local/bytes_read_from_remote
   - prevent inflated FileCache profile counters caused by repeated accumulation (e.g. LockWaitTimer, CacheGetOrSetTimer, BytesWriteIntoCache)
   
   ### What problem does this PR solve?
   
   Issue Number: close #xxx
   
   Related PR: #xxx
   
   Problem Summary:
   
   ### Release note
   
   None
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [ ] Regression test
       - [ ] Unit Test
       - [ ] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
   - Behavior changed:
       - [ ] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [ ] No.
       - [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #61072:
URL: https://github.com/apache/doris/pull/61072#discussion_r3285796579


##########
be/src/exec/scan/olap_scanner.cpp:
##########
@@ -681,25 +681,30 @@ void OlapScanner::update_realtime_counters() {
                 stats.compressed_bytes_read);
     } else {
         _state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         _state->get_query_ctx()
                 ->resource_ctx()
                 ->io_context()
                 ->update_scan_bytes_from_remote_storage(
-                        stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                        stats.file_cache_stats.bytes_read_from_remote);
 
         DorisMetrics::instance()->query_scan_bytes_from_local->increment(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
-                stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                stats.file_cache_stats.bytes_read_from_remote);
+    }
+
+    if (config::is_cloud_mode() && config::enable_file_cache) {

Review Comment:
   this check will ignore cold-hot separation, which may use file cache too
   is that expected?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4003235282

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #61072:
URL: https://github.com/apache/doris/pull/61072


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4003235166

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and after the optimization?
       


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4089460273

   # BE Regression && UT Coverage Report
   Increment line coverage `100% (0/0)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.39% (26959/36735) |
   | Line Coverage     | 56.86% (288937/508156) |
   | Region Coverage   | 54.09% (240324/444311) |
   | Branch Coverage   | 55.85% (104041/186279) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4140540824

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4301963730

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (11/11)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.53% (27513/37418) |
   | Line Coverage     | 57.31% (297581/519225) |
   | Region Coverage   | 54.48% (247344/454029) |
   | Branch Coverage   | 56.07% (107129/191070) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4599540127

   # BE UT Coverage Report
   Increment line coverage `0.00% (0/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/7032373e9a616e61893b6a8b067deb91b26a267b_7032373e9a616e61893b6a8b067deb91b26a267b/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/7032373e9a616e61893b6a8b067deb91b26a267b_7032373e9a616e61893b6a8b067deb91b26a267b/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 54.13% (21107/38993) |
   | Line Coverage     | 37.62% (200041/531741) |
   | Region Coverage   | 33.86% (156746/462936) |
   | Branch Coverage   | 34.81% (68097/195633) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4598514567

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4003449840

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 27596 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 9f9f189378b0b0963c69a19d0697dab1f296dee0, data reload: false
   
   ------ Round 1 ----------------------------------
   ============================================
   q1	17673	4454	4324	4324
   q2	q3	10663	793	512	512
   q4	4689	359	252	252
   q5	7547	1178	1018	1018
   q6	174	174	153	153
   q7	762	847	643	643
   q8	9288	1445	1283	1283
   q9	4905	4751	4673	4673
   q10	6327	1869	1648	1648
   q11	462	242	243	242
   q12	756	563	454	454
   q13	18072	2912	2179	2179
   q14	230	221	220	220
   q15	941	809	799	799
   q16	742	721	653	653
   q17	684	821	447	447
   q18	5914	5444	5273	5273
   q19	1240	979	599	599
   q20	490	476	393	393
   q21	4418	1987	1546	1546
   q22	356	292	285	285
   Total cold run time: 96333 ms
   Total hot run time: 27596 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   ============================================
   q1	4750	4597	4519	4519
   q2	q3	3869	4360	3865	3865
   q4	869	1203	773	773
   q5	4053	4365	4421	4365
   q6	178	178	142	142
   q7	1798	1640	1547	1547
   q8	2471	2642	2616	2616
   q9	7513	7343	7452	7343
   q10	3752	3944	3602	3602
   q11	495	432	407	407
   q12	486	594	436	436
   q13	2657	3319	2314	2314
   q14	286	321	285	285
   q15	851	805	789	789
   q16	749	772	718	718
   q17	1187	1514	1330	1330
   q18	7172	6776	6653	6653
   q19	878	815	845	815
   q20	2058	2131	2066	2066
   q21	3915	3449	3343	3343
   q22	463	429	398	398
   Total cold run time: 50450 ms
   Total hot run time: 48326 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4088719221

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4088950851

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 168073 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf, data reload: false
   
   query5	4333	641	492	492
   query6	339	226	212	212
   query7	4231	466	265	265
   query8	352	250	239	239
   query9	8741	2696	2722	2696
   query10	507	406	358	358
   query11	7008	5064	4845	4845
   query12	185	135	125	125
   query13	1277	471	356	356
   query14	5773	3731	3451	3451
   query14_1	2832	2881	2847	2847
   query15	204	192	174	174
   query16	979	489	452	452
   query17	1125	730	637	637
   query18	2443	461	362	362
   query19	245	204	178	178
   query20	139	128	126	126
   query21	215	137	110	110
   query22	13164	14002	14718	14002
   query23	16240	15723	15759	15723
   query23_1	15613	15614	15173	15173
   query24	7157	1590	1210	1210
   query24_1	1223	1226	1217	1217
   query25	543	465	407	407
   query26	1229	254	145	145
   query27	2784	475	293	293
   query28	4486	1811	1810	1810
   query29	863	587	463	463
   query30	296	217	192	192
   query31	1013	935	881	881
   query32	85	69	73	69
   query33	510	335	277	277
   query34	897	864	523	523
   query35	645	677	607	607
   query36	1082	1143	996	996
   query37	131	93	84	84
   query38	2908	2914	2864	2864
   query39	875	845	810	810
   query39_1	803	799	787	787
   query40	236	151	136	136
   query41	60	60	58	58
   query42	260	256	254	254
   query43	238	247	213	213
   query44	
   query45	196	189	183	183
   query46	873	965	599	599
   query47	2100	2143	2063	2063
   query48	302	317	234	234
   query49	626	470	384	384
   query50	681	279	218	218
   query51	4025	4013	4042	4013
   query52	263	264	258	258
   query53	290	334	285	285
   query54	298	268	264	264
   query55	90	85	85	85
   query56	323	314	345	314
   query57	1948	1699	1714	1699
   query58	288	275	273	273
   query59	2806	2926	2758	2758
   query60	349	335	323	323
   query61	152	153	154	153
   query62	646	596	533	533
   query63	316	283	284	283
   query64	5105	1266	1014	1014
   query65	
   query66	1501	449	357	357
   query67	24415	24215	24133	24133
   query68	
   query69	411	330	289	289
   query70	995	948	919	919
   query71	343	310	307	307
   query72	2728	2766	2596	2596
   query73	543	553	313	313
   query74	9576	9593	9403	9403
   query75	2835	2754	2478	2478
   query76	2299	1051	688	688
   query77	389	396	335	335
   query78	10983	11209	10498	10498
   query79	1097	760	575	575
   query80	746	676	591	591
   query81	479	259	224	224
   query82	1351	152	122	122
   query83	379	269	248	248
   query84	256	121	106	106
   query85	910	503	445	445
   query86	360	322	300	300
   query87	3170	3086	2988	2988
   query88	3545	2653	2662	2653
   query89	430	369	342	342
   query90	1982	179	181	179
   query91	177	166	143	143
   query92	76	78	72	72
   query93	941	839	495	495
   query94	468	332	289	289
   query95	587	333	389	333
   query96	648	538	228	228
   query97	2480	2500	2429	2429
   query98	237	222	219	219
   query99	970	993	904	904
   Total cold run time: 248355 ms
   Total hot run time: 168073 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4088889442

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 26824 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17685	4483	4301	4301
   q2	q3	10646	817	523	523
   q4	4674	352	249	249
   q5	7561	1203	1009	1009
   q6	178	176	147	147
   q7	789	863	662	662
   q8	9314	1519	1321	1321
   q9	4903	4697	4710	4697
   q10	6258	1923	1650	1650
   q11	483	276	239	239
   q12	713	583	471	471
   q13	18029	2917	2179	2179
   q14	234	237	209	209
   q15	q16	719	752	663	663
   q17	719	831	467	467
   q18	5983	5457	5347	5347
   q19	1244	1004	606	606
   q20	533	481	376	376
   q21	4645	1841	1427	1427
   q22	555	362	281	281
   Total cold run time: 95865 ms
   Total hot run time: 26824 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	4787	4549	4570	4549
   q2	q3	3925	4354	3852	3852
   q4	877	1231	763	763
   q5	4068	4378	4334	4334
   q6	186	176	140	140
   q7	1779	1662	1544	1544
   q8	2546	2718	2538	2538
   q9	7809	7229	7290	7229
   q10	3782	4020	3571	3571
   q11	499	432	412	412
   q12	495	575	447	447
   q13	2721	3128	2346	2346
   q14	276	318	268	268
   q15	q16	719	750	775	750
   q17	1173	1421	1376	1376
   q18	7139	6863	6692	6692
   q19	903	895	906	895
   q20	2063	2191	2027	2027
   q21	3866	3520	3495	3495
   q22	478	428	382	382
   Total cold run time: 50091 ms
   Total hot run time: 47610 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4089195889

   # BE UT Coverage Report
   Increment line coverage ` ` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf_7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf_7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 52.70% (19780/37532) |
   | Line Coverage     | 36.24% (184784/509870) |
   | Region Coverage   | 32.48% (142997/440265) |
   | Branch Coverage   | 33.66% (62542/185797) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61072:
URL: https://github.com/apache/doris/pull/61072#discussion_r2894324189


##########
be/src/vec/exec/scan/olap_scanner.cpp:
##########
@@ -692,25 +692,31 @@ void OlapScanner::update_realtime_counters() {
                 stats.compressed_bytes_read);
     } else {
         _state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         _state->get_query_ctx()
                 ->resource_ctx()
                 ->io_context()
                 ->update_scan_bytes_from_remote_storage(
-                        stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                        stats.file_cache_stats.bytes_read_from_remote);
 
         DorisMetrics::instance()->query_scan_bytes_from_local->increment(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
-                stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                stats.file_cache_stats.bytes_read_from_remote);
+    }
+
+    if (config::is_cloud_mode() && config::enable_file_cache) {
+        io::FileCacheProfileReporter cache_profile(local_state->_segment_profile.get());
+        cache_profile.update(&stats.file_cache_stats);
     }
 
     _tablet_reader->mutable_stats()->compressed_bytes_read = 0;
     _tablet_reader->mutable_stats()->uncompressed_bytes_read = 0;
     _tablet_reader->mutable_stats()->raw_rows_read = 0;
+    _tablet_reader->mutable_stats()->file_cache_stats = {};
 
-    _bytes_read_from_local = _tablet_reader->stats().file_cache_stats.bytes_read_from_local;
-    _bytes_read_from_remote = _tablet_reader->stats().file_cache_stats.bytes_read_from_remote;
+    _bytes_read_from_local = 0;

Review Comment:
   nit (dead code): After this PR, `_bytes_read_from_local` and `_bytes_read_from_remote` are effectively dead code — they are declared (initialized to 0 in `olap_scanner.h:106-107`), and only ever assigned 0 here. They are never read anywhere.
   
   Consider removing these two member variables from the header and these two assignment lines entirely. This would simplify the code and avoid future confusion about their purpose.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4004217984

   # BE UT Coverage Report
   Increment line coverage `0.00% (0/11)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/9f9f189378b0b0963c69a19d0697dab1f296dee0_9f9f189378b0b0963c69a19d0697dab1f296dee0/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/9f9f189378b0b0963c69a19d0697dab1f296dee0_9f9f189378b0b0963c69a19d0697dab1f296dee0/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 52.56% (19646/37381) |
   | Line Coverage     | 36.18% (183425/506991) |
   | Region Coverage   | 32.49% (142343/438104) |
   | Branch Coverage   | 33.43% (61736/184662) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4598583939

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4599681735

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (31/31)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_7032373e9a616e61893b6a8b067deb91b26a267b_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_7032373e9a616e61893b6a8b067deb91b26a267b_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.92% (28222/38181) |
   | Line Coverage     | 57.83% (306729/530363) |
   | Region Coverage   | 54.97% (256911/467355) |
   | Branch Coverage   | 56.50% (110950/196379) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4140540921

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #61072:
URL: https://github.com/apache/doris/pull/61072#discussion_r2894319292


##########
be/src/vec/exec/scan/olap_scanner.cpp:
##########
@@ -692,25 +692,31 @@ void OlapScanner::update_realtime_counters() {
                 stats.compressed_bytes_read);
     } else {
         _state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         _state->get_query_ctx()
                 ->resource_ctx()
                 ->io_context()
                 ->update_scan_bytes_from_remote_storage(
-                        stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                        stats.file_cache_stats.bytes_read_from_remote);
 
         DorisMetrics::instance()->query_scan_bytes_from_local->increment(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
-                stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                stats.file_cache_stats.bytes_read_from_remote);
+    }
+
+    if (config::is_cloud_mode() && config::enable_file_cache) {
+        io::FileCacheProfileReporter cache_profile(local_state->_segment_profile.get());
+        cache_profile.update(&stats.file_cache_stats);
     }
 
     _tablet_reader->mutable_stats()->compressed_bytes_read = 0;
     _tablet_reader->mutable_stats()->uncompressed_bytes_read = 0;
     _tablet_reader->mutable_stats()->raw_rows_read = 0;
+    _tablet_reader->mutable_stats()->file_cache_stats = {};

Review Comment:
   Bug: By resetting `file_cache_stats = {}` at line 716, the cumulative `bytes_write_into_cache` that was previously tracked across all cycles is now lost. In `_collect_profile_before_close()` (line 846–847), `update_bytes_write_into_cache` is called with `stats.file_cache_stats.bytes_write_into_cache`, but since `file_cache_stats` is now reset to `{}` every cycle, this will only report the bytes written since the last `update_realtime_counters()` call, not the total.
   
   To fix this, you should also call `update_bytes_write_into_cache` inside `update_realtime_counters` (in the `config::is_cloud_mode() && config::enable_file_cache` block), similar to how `cache_profile.update` is called here.



##########
be/src/vec/exec/scan/olap_scanner.cpp:
##########
@@ -692,25 +692,31 @@ void OlapScanner::update_realtime_counters() {
                 stats.compressed_bytes_read);
     } else {
         _state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         _state->get_query_ctx()
                 ->resource_ctx()
                 ->io_context()
                 ->update_scan_bytes_from_remote_storage(
-                        stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                        stats.file_cache_stats.bytes_read_from_remote);
 
         DorisMetrics::instance()->query_scan_bytes_from_local->increment(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
-                stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                stats.file_cache_stats.bytes_read_from_remote);
+    }
+
+    if (config::is_cloud_mode() && config::enable_file_cache) {
+        io::FileCacheProfileReporter cache_profile(local_state->_segment_profile.get());
+        cache_profile.update(&stats.file_cache_stats);
     }
 
     _tablet_reader->mutable_stats()->compressed_bytes_read = 0;
     _tablet_reader->mutable_stats()->uncompressed_bytes_read = 0;
     _tablet_reader->mutable_stats()->raw_rows_read = 0;
+    _tablet_reader->mutable_stats()->file_cache_stats = {};
 
-    _bytes_read_from_local = _tablet_reader->stats().file_cache_stats.bytes_read_from_local;
-    _bytes_read_from_remote = _tablet_reader->stats().file_cache_stats.bytes_read_from_remote;
+    _bytes_read_from_local = 0;
+    _bytes_read_from_remote = 0;

Review Comment:
   The member variables `_bytes_read_from_local` and `_bytes_read_from_remote` are now dead code. Since `file_cache_stats` is reset to `{}` at the end of each cycle, these variables are always set to 0 at lines 718-719 and never read before being re-assigned. They should be removed from both the `.cpp` and `.h` files to avoid confusion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4301553777

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4302892139

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (11/11)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 73.53% (27513/37418) |
   | Line Coverage     | 57.31% (297581/519225) |
   | Region Coverage   | 54.48% (247344/454029) |
   | Branch Coverage   | 56.07% (107129/191070) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4010105659

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4003514699

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 153118 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 9f9f189378b0b0963c69a19d0697dab1f296dee0, data reload: false
   
   query5	4326	667	520	520
   query6	327	228	204	204
   query7	4210	487	267	267
   query8	328	239	225	225
   query9	8718	2751	2732	2732
   query10	523	412	331	331
   query11	7338	5877	5533	5533
   query12	179	136	125	125
   query13	1261	470	373	373
   query14	5778	3833	3538	3538
   query14_1	2815	2828	2828	2828
   query15	204	197	176	176
   query16	1014	459	422	422
   query17	1073	697	597	597
   query18	2432	442	342	342
   query19	214	206	177	177
   query20	140	129	128	128
   query21	225	148	117	117
   query22	4904	5058	5020	5020
   query23	16635	16059	15633	15633
   query23_1	15875	15873	16070	15873
   query24	7534	1727	1285	1285
   query24_1	1280	1261	1275	1261
   query25	664	539	496	496
   query26	1280	298	176	176
   query27	2769	499	289	289
   query28	4480	1897	1875	1875
   query29	873	607	495	495
   query30	311	253	213	213
   query31	1371	1319	1220	1220
   query32	85	76	72	72
   query33	518	354	289	289
   query34	927	981	560	560
   query35	662	698	614	614
   query36	1105	1156	951	951
   query37	147	97	86	86
   query38	2949	2941	2841	2841
   query39	891	867	851	851
   query39_1	824	881	834	834
   query40	241	163	144	144
   query41	67	66	69	66
   query42	319	311	305	305
   query43	247	250	225	225
   query44	
   query45	196	187	181	181
   query46	920	1026	614	614
   query47	2132	2138	2059	2059
   query48	340	337	233	233
   query49	633	481	391	391
   query50	712	285	214	214
   query51	4156	4141	4071	4071
   query52	286	295	292	292
   query53	295	351	286	286
   query54	312	287	280	280
   query55	91	88	87	87
   query56	332	326	315	315
   query57	1363	1319	1293	1293
   query58	303	286	280	280
   query59	1313	1431	1294	1294
   query60	353	337	317	317
   query61	144	138	146	138
   query62	632	590	543	543
   query63	322	281	276	276
   query64	5068	1277	980	980
   query65	
   query66	1443	493	353	353
   query67	16230	16423	16313	16313
   query68	
   query69	394	309	332	309
   query70	975	923	942	923
   query71	347	307	288	288
   query72	2781	2735	2432	2432
   query73	557	585	327	327
   query74	9990	9901	9781	9781
   query75	2948	2776	2458	2458
   query76	2281	1133	696	696
   query77	384	407	301	301
   query78	11209	11281	10704	10704
   query79	3134	860	603	603
   query80	1731	642	532	532
   query81	594	279	243	243
   query82	1019	150	117	117
   query83	324	265	245	245
   query84	254	118	96	96
   query85	904	485	427	427
   query86	453	336	295	295
   query87	3123	3160	2986	2986
   query88	3588	2636	2639	2636
   query89	457	374	340	340
   query90	1985	183	171	171
   query91	162	156	133	133
   query92	78	72	76	72
   query93	1387	844	511	511
   query94	648	313	282	282
   query95	581	408	312	312
   query96	657	543	227	227
   query97	2550	2554	2436	2436
   query98	234	237	217	217
   query99	1031	987	918	918
   Total cold run time: 237453 ms
   Total hot run time: 153118 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4600435659

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4088803234

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4094879559

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4004497073

   # BE Regression && UT Coverage Report
   Increment line coverage `100.00% (11/11)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_9f9f189378b0b0963c69a19d0697dab1f296dee0_merge/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_9f9f189378b0b0963c69a19d0697dab1f296dee0_merge/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 71.49% (26166/36603) |
   | Line Coverage     | 54.25% (274195/505438) |
   | Region Coverage   | 51.58% (228089/442241) |
   | Branch Coverage   | 52.88% (97945/185226) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4301773625

   # BE UT Coverage Report
   Increment line coverage `0.00% (0/11)` :tada:
   
   [Increment coverage report](http://coverage.selectdb-in.cc/coverage/1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d/increment_report/index.html)
   [Complete coverage report](http://coverage.selectdb-in.cc/coverage/1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d/report/index.html)
   | Category          | Coverage           |
   |-------------------|--------------------|
   | Function Coverage | 53.33% (20367/38193) |
   | Line Coverage     | 36.87% (191898/520520) |
   | Region Coverage   | 33.20% (149323/449703) |
   | Branch Coverage   | 34.29% (65266/190340) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4302288067

   run cloud_p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #61072:
URL: https://github.com/apache/doris/pull/61072#discussion_r3338538019


##########
be/src/exec/scan/olap_scanner.cpp:
##########
@@ -681,25 +681,30 @@ void OlapScanner::update_realtime_counters() {
                 stats.compressed_bytes_read);
     } else {
         _state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         _state->get_query_ctx()
                 ->resource_ctx()
                 ->io_context()
                 ->update_scan_bytes_from_remote_storage(
-                        stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                        stats.file_cache_stats.bytes_read_from_remote);
 
         DorisMetrics::instance()->query_scan_bytes_from_local->increment(
-                stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+                stats.file_cache_stats.bytes_read_from_local);
         DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
-                stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+                stats.file_cache_stats.bytes_read_from_remote);
+    }
+
+    if (config::is_cloud_mode() && config::enable_file_cache) {

Review Comment:
   done fix



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4602074532

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 29727 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 7032373e9a616e61893b6a8b067deb91b26a267b, data reload: false
   
   ------ Round 1 ----------------------------------
   orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	17742	4184	4101	4101
   q2	q3	10747	1427	831	831
   q4	4688	487	348	348
   q5	7538	911	601	601
   q6	191	175	137	137
   q7	785	844	663	663
   q8	9365	1683	1653	1653
   q9	5905	4491	4542	4491
   q10	6785	1854	1546	1546
   q11	441	280	258	258
   q12	642	424	289	289
   q13	18173	3537	2826	2826
   q14	270	259	243	243
   q15	q16	821	879	715	715
   q17	971	963	921	921
   q18	7238	5784	5652	5652
   q19	1331	1329	1068	1068
   q20	518	401	261	261
   q21	6183	2843	2799	2799
   q22	461	389	324	324
   Total cold run time: 100795 ms
   Total hot run time: 29727 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
   ============================================
   q1	5168	4891	4893	4891
   q2	q3	4932	5355	4661	4661
   q4	2150	2219	1430	1430
   q5	4938	4958	4758	4758
   q6	240	177	128	128
   q7	1844	1790	1634	1634
   q8	2523	2259	2194	2194
   q9	8080	7854	7482	7482
   q10	4767	4676	4196	4196
   q11	547	382	352	352
   q12	745	736	525	525
   q13	3070	3394	2846	2846
   q14	268	285	252	252
   q15	q16	680	697	613	613
   q17	1305	1277	1259	1259
   q18	7388	6812	6761	6761
   q19	1106	1089	1123	1089
   q20	2227	2220	1969	1969
   q21	5367	4673	4512	4512
   q22	535	456	415	415
   Total cold run time: 57880 ms
   Total hot run time: 51967 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]

Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4602155463

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 168982 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 7032373e9a616e61893b6a8b067deb91b26a267b, data reload: false
   
   query5	4332	631	495	495
   query6	448	199	183	183
   query7	4804	564	305	305
   query8	373	224	215	215
   query9	8754	3984	3983	3983
   query10	445	316	260	260
   query11	5926	2392	2165	2165
   query12	153	100	96	96
   query13	1272	618	419	419
   query14	6451	5406	5084	5084
   query14_1	4430	4420	4386	4386
   query15	204	200	175	175
   query16	1007	446	437	437
   query17	1110	690	563	563
   query18	2455	481	347	347
   query19	196	183	143	143
   query20	110	106	109	106
   query21	218	140	119	119
   query22	13679	13506	13287	13287
   query23	17507	16532	16204	16204
   query23_1	16265	16187	16335	16187
   query24	7754	1779	1324	1324
   query24_1	1316	1311	1313	1311
   query25	547	450	375	375
   query26	1294	320	168	168
   query27	2695	542	343	343
   query28	4465	2031	1984	1984
   query29	1087	610	500	500
   query30	312	235	195	195
   query31	1118	1072	939	939
   query32	103	60	61	60
   query33	529	318	261	261
   query34	1182	1141	652	652
   query35	776	794	661	661
   query36	1424	1378	1235	1235
   query37	155	104	94	94
   query38	3168	3088	3057	3057
   query39	937	913	892	892
   query39_1	871	880	877	877
   query40	230	120	99	99
   query41	64	61	63	61
   query42	96	92	94	92
   query43	318	324	278	278
   query44	
   query45	196	186	178	178
   query46	1108	1196	754	754
   query47	2357	2395	2265	2265
   query48	407	413	302	302
   query49	658	469	346	346
   query50	969	356	258	258
   query51	4283	4305	4234	4234
   query52	87	90	76	76
   query53	256	273	197	197
   query54	274	219	192	192
   query55	77	78	69	69
   query56	235	227	229	227
   query57	1446	1398	1301	1301
   query58	244	215	228	215
   query59	1585	1669	1460	1460
   query60	283	263	225	225
   query61	164	158	164	158
   query62	698	646	575	575
   query63	236	183	187	183
   query64	2642	875	689	689
   query65	
   query66	1846	501	363	363
   query67	29697	29603	29573	29573
   query68	
   query69	431	314	277	277
   query70	965	995	953	953
   query71	294	225	205	205
   query72	3015	2695	2397	2397
   query73	869	788	447	447
   query74	5134	4967	4791	4791
   query75	2652	2572	2246	2246
   query76	2299	1153	785	785
   query77	348	389	286	286
   query78	12347	12417	11860	11860
   query79	1335	1087	767	767
   query80	525	467	380	380
   query81	446	287	234	234
   query82	232	158	130	130
   query83	298	275	249	249
   query84	263	145	115	115
   query85	848	530	445	445
   query86	321	302	278	278
   query87	3339	3346	3208	3208
   query88	3617	2732	2731	2731
   query89	416	385	328	328
   query90	2173	190	182	182
   query91	180	170	135	135
   query92	63	57	58	57
   query93	1445	1556	849	849
   query94	522	362	315	315
   query95	684	466	361	361
   query96	1053	794	328	328
   query97	2712	2699	2592	2592
   query98	212	209	205	205
   query99	1141	1174	1040	1040
   Total cold run time: 251035 ms
   Total hot run time: 168982 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org