You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "freemandealer (via GitHub)" <gi...@apache.org> on 2026/03/05 08:23:50 UTC
[PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
freemandealer opened a new pull request, #61072:
URL: https://github.com/apache/doris/pull/61072
- decouple realtime FileCache profile updates from the local/remote-bytes branch and update once per cycle when file cache is enabled
- reset file_cache_stats as a whole (`= {}`) after realtime reporting, instead of resetting only bytes_read_from_local/bytes_read_from_remote
- prevent inflated FileCache profile counters caused by repeated accumulation (e.g. LockWaitTimer, CacheGetOrSetTimer, BytesWriteIntoCache)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #61072:
URL: https://github.com/apache/doris/pull/61072#discussion_r3285796579
##########
be/src/exec/scan/olap_scanner.cpp:
##########
@@ -681,25 +681,30 @@ void OlapScanner::update_realtime_counters() {
stats.compressed_bytes_read);
} else {
_state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
_state->get_query_ctx()
->resource_ctx()
->io_context()
->update_scan_bytes_from_remote_storage(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
DorisMetrics::instance()->query_scan_bytes_from_local->increment(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
+ }
+
+ if (config::is_cloud_mode() && config::enable_file_cache) {
Review Comment:
this check will ignore cold-hot separation, which may use file cache too
is that expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4003235282
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #61072:
URL: https://github.com/apache/doris/pull/61072
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4003235166
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4089460273
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.39% (26959/36735) |
| Line Coverage | 56.86% (288937/508156) |
| Region Coverage | 54.09% (240324/444311) |
| Branch Coverage | 55.85% (104041/186279) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4140540824
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4301963730
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (11/11)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.53% (27513/37418) |
| Line Coverage | 57.31% (297581/519225) |
| Region Coverage | 54.48% (247344/454029) |
| Branch Coverage | 56.07% (107129/191070) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4599540127
# BE UT Coverage Report
Increment line coverage `0.00% (0/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/7032373e9a616e61893b6a8b067deb91b26a267b_7032373e9a616e61893b6a8b067deb91b26a267b/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/7032373e9a616e61893b6a8b067deb91b26a267b_7032373e9a616e61893b6a8b067deb91b26a267b/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 54.13% (21107/38993) |
| Line Coverage | 37.62% (200041/531741) |
| Region Coverage | 33.86% (156746/462936) |
| Branch Coverage | 34.81% (68097/195633) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4598514567
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4003449840
<details>
<summary>TPC-H: <b>Total hot run time: 27596 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f9f189378b0b0963c69a19d0697dab1f296dee0, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17673 4454 4324 4324
q2 q3 10663 793 512 512
q4 4689 359 252 252
q5 7547 1178 1018 1018
q6 174 174 153 153
q7 762 847 643 643
q8 9288 1445 1283 1283
q9 4905 4751 4673 4673
q10 6327 1869 1648 1648
q11 462 242 243 242
q12 756 563 454 454
q13 18072 2912 2179 2179
q14 230 221 220 220
q15 941 809 799 799
q16 742 721 653 653
q17 684 821 447 447
q18 5914 5444 5273 5273
q19 1240 979 599 599
q20 490 476 393 393
q21 4418 1987 1546 1546
q22 356 292 285 285
Total cold run time: 96333 ms
Total hot run time: 27596 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4750 4597 4519 4519
q2 q3 3869 4360 3865 3865
q4 869 1203 773 773
q5 4053 4365 4421 4365
q6 178 178 142 142
q7 1798 1640 1547 1547
q8 2471 2642 2616 2616
q9 7513 7343 7452 7343
q10 3752 3944 3602 3602
q11 495 432 407 407
q12 486 594 436 436
q13 2657 3319 2314 2314
q14 286 321 285 285
q15 851 805 789 789
q16 749 772 718 718
q17 1187 1514 1330 1330
q18 7172 6776 6653 6653
q19 878 815 845 815
q20 2058 2131 2066 2066
q21 3915 3449 3343 3343
q22 463 429 398 398
Total cold run time: 50450 ms
Total hot run time: 48326 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4088719221
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4088950851
<details>
<summary>TPC-DS: <b>Total hot run time: 168073 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf, data reload: false
query5 4333 641 492 492
query6 339 226 212 212
query7 4231 466 265 265
query8 352 250 239 239
query9 8741 2696 2722 2696
query10 507 406 358 358
query11 7008 5064 4845 4845
query12 185 135 125 125
query13 1277 471 356 356
query14 5773 3731 3451 3451
query14_1 2832 2881 2847 2847
query15 204 192 174 174
query16 979 489 452 452
query17 1125 730 637 637
query18 2443 461 362 362
query19 245 204 178 178
query20 139 128 126 126
query21 215 137 110 110
query22 13164 14002 14718 14002
query23 16240 15723 15759 15723
query23_1 15613 15614 15173 15173
query24 7157 1590 1210 1210
query24_1 1223 1226 1217 1217
query25 543 465 407 407
query26 1229 254 145 145
query27 2784 475 293 293
query28 4486 1811 1810 1810
query29 863 587 463 463
query30 296 217 192 192
query31 1013 935 881 881
query32 85 69 73 69
query33 510 335 277 277
query34 897 864 523 523
query35 645 677 607 607
query36 1082 1143 996 996
query37 131 93 84 84
query38 2908 2914 2864 2864
query39 875 845 810 810
query39_1 803 799 787 787
query40 236 151 136 136
query41 60 60 58 58
query42 260 256 254 254
query43 238 247 213 213
query44
query45 196 189 183 183
query46 873 965 599 599
query47 2100 2143 2063 2063
query48 302 317 234 234
query49 626 470 384 384
query50 681 279 218 218
query51 4025 4013 4042 4013
query52 263 264 258 258
query53 290 334 285 285
query54 298 268 264 264
query55 90 85 85 85
query56 323 314 345 314
query57 1948 1699 1714 1699
query58 288 275 273 273
query59 2806 2926 2758 2758
query60 349 335 323 323
query61 152 153 154 153
query62 646 596 533 533
query63 316 283 284 283
query64 5105 1266 1014 1014
query65
query66 1501 449 357 357
query67 24415 24215 24133 24133
query68
query69 411 330 289 289
query70 995 948 919 919
query71 343 310 307 307
query72 2728 2766 2596 2596
query73 543 553 313 313
query74 9576 9593 9403 9403
query75 2835 2754 2478 2478
query76 2299 1051 688 688
query77 389 396 335 335
query78 10983 11209 10498 10498
query79 1097 760 575 575
query80 746 676 591 591
query81 479 259 224 224
query82 1351 152 122 122
query83 379 269 248 248
query84 256 121 106 106
query85 910 503 445 445
query86 360 322 300 300
query87 3170 3086 2988 2988
query88 3545 2653 2662 2653
query89 430 369 342 342
query90 1982 179 181 179
query91 177 166 143 143
query92 76 78 72 72
query93 941 839 495 495
query94 468 332 289 289
query95 587 333 389 333
query96 648 538 228 228
query97 2480 2500 2429 2429
query98 237 222 219 219
query99 970 993 904 904
Total cold run time: 248355 ms
Total hot run time: 168073 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4088889442
<details>
<summary>TPC-H: <b>Total hot run time: 26824 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17685 4483 4301 4301
q2 q3 10646 817 523 523
q4 4674 352 249 249
q5 7561 1203 1009 1009
q6 178 176 147 147
q7 789 863 662 662
q8 9314 1519 1321 1321
q9 4903 4697 4710 4697
q10 6258 1923 1650 1650
q11 483 276 239 239
q12 713 583 471 471
q13 18029 2917 2179 2179
q14 234 237 209 209
q15 q16 719 752 663 663
q17 719 831 467 467
q18 5983 5457 5347 5347
q19 1244 1004 606 606
q20 533 481 376 376
q21 4645 1841 1427 1427
q22 555 362 281 281
Total cold run time: 95865 ms
Total hot run time: 26824 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4787 4549 4570 4549
q2 q3 3925 4354 3852 3852
q4 877 1231 763 763
q5 4068 4378 4334 4334
q6 186 176 140 140
q7 1779 1662 1544 1544
q8 2546 2718 2538 2538
q9 7809 7229 7290 7229
q10 3782 4020 3571 3571
q11 499 432 412 412
q12 495 575 447 447
q13 2721 3128 2346 2346
q14 276 318 268 268
q15 q16 719 750 775 750
q17 1173 1421 1376 1376
q18 7139 6863 6692 6692
q19 903 895 906 895
q20 2063 2191 2027 2027
q21 3866 3520 3495 3495
q22 478 428 382 382
Total cold run time: 50091 ms
Total hot run time: 47610 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4089195889
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf_7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf_7cd6e2e3395e460ddfc3f6d696f8832a9ecd0bdf/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.70% (19780/37532) |
| Line Coverage | 36.24% (184784/509870) |
| Region Coverage | 32.48% (142997/440265) |
| Branch Coverage | 33.66% (62542/185797) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #61072:
URL: https://github.com/apache/doris/pull/61072#discussion_r2894324189
##########
be/src/vec/exec/scan/olap_scanner.cpp:
##########
@@ -692,25 +692,31 @@ void OlapScanner::update_realtime_counters() {
stats.compressed_bytes_read);
} else {
_state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
_state->get_query_ctx()
->resource_ctx()
->io_context()
->update_scan_bytes_from_remote_storage(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
DorisMetrics::instance()->query_scan_bytes_from_local->increment(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
+ }
+
+ if (config::is_cloud_mode() && config::enable_file_cache) {
+ io::FileCacheProfileReporter cache_profile(local_state->_segment_profile.get());
+ cache_profile.update(&stats.file_cache_stats);
}
_tablet_reader->mutable_stats()->compressed_bytes_read = 0;
_tablet_reader->mutable_stats()->uncompressed_bytes_read = 0;
_tablet_reader->mutable_stats()->raw_rows_read = 0;
+ _tablet_reader->mutable_stats()->file_cache_stats = {};
- _bytes_read_from_local = _tablet_reader->stats().file_cache_stats.bytes_read_from_local;
- _bytes_read_from_remote = _tablet_reader->stats().file_cache_stats.bytes_read_from_remote;
+ _bytes_read_from_local = 0;
Review Comment:
nit (dead code): After this PR, `_bytes_read_from_local` and `_bytes_read_from_remote` are effectively dead code — they are declared (initialized to 0 in `olap_scanner.h:106-107`), and only ever assigned 0 here. They are never read anywhere.
Consider removing these two member variables from the header and these two assignment lines entirely. This would simplify the code and avoid future confusion about their purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4004217984
# BE UT Coverage Report
Increment line coverage `0.00% (0/11)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/9f9f189378b0b0963c69a19d0697dab1f296dee0_9f9f189378b0b0963c69a19d0697dab1f296dee0/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/9f9f189378b0b0963c69a19d0697dab1f296dee0_9f9f189378b0b0963c69a19d0697dab1f296dee0/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.56% (19646/37381) |
| Line Coverage | 36.18% (183425/506991) |
| Region Coverage | 32.49% (142343/438104) |
| Branch Coverage | 33.43% (61736/184662) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4598583939
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4599681735
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (31/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_7032373e9a616e61893b6a8b067deb91b26a267b_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_7032373e9a616e61893b6a8b067deb91b26a267b_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.92% (28222/38181) |
| Line Coverage | 57.83% (306729/530363) |
| Region Coverage | 54.97% (256911/467355) |
| Branch Coverage | 56.50% (110950/196379) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4140540921
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #61072:
URL: https://github.com/apache/doris/pull/61072#discussion_r2894319292
##########
be/src/vec/exec/scan/olap_scanner.cpp:
##########
@@ -692,25 +692,31 @@ void OlapScanner::update_realtime_counters() {
stats.compressed_bytes_read);
} else {
_state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
_state->get_query_ctx()
->resource_ctx()
->io_context()
->update_scan_bytes_from_remote_storage(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
DorisMetrics::instance()->query_scan_bytes_from_local->increment(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
+ }
+
+ if (config::is_cloud_mode() && config::enable_file_cache) {
+ io::FileCacheProfileReporter cache_profile(local_state->_segment_profile.get());
+ cache_profile.update(&stats.file_cache_stats);
}
_tablet_reader->mutable_stats()->compressed_bytes_read = 0;
_tablet_reader->mutable_stats()->uncompressed_bytes_read = 0;
_tablet_reader->mutable_stats()->raw_rows_read = 0;
+ _tablet_reader->mutable_stats()->file_cache_stats = {};
Review Comment:
Bug: By resetting `file_cache_stats = {}` at line 716, the cumulative `bytes_write_into_cache` that was previously tracked across all cycles is now lost. In `_collect_profile_before_close()` (line 846–847), `update_bytes_write_into_cache` is called with `stats.file_cache_stats.bytes_write_into_cache`, but since `file_cache_stats` is now reset to `{}` every cycle, this will only report the bytes written since the last `update_realtime_counters()` call, not the total.
To fix this, you should also call `update_bytes_write_into_cache` inside `update_realtime_counters` (in the `config::is_cloud_mode() && config::enable_file_cache` block), similar to how `cache_profile.update` is called here.
##########
be/src/vec/exec/scan/olap_scanner.cpp:
##########
@@ -692,25 +692,31 @@ void OlapScanner::update_realtime_counters() {
stats.compressed_bytes_read);
} else {
_state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
_state->get_query_ctx()
->resource_ctx()
->io_context()
->update_scan_bytes_from_remote_storage(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
DorisMetrics::instance()->query_scan_bytes_from_local->increment(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
+ }
+
+ if (config::is_cloud_mode() && config::enable_file_cache) {
+ io::FileCacheProfileReporter cache_profile(local_state->_segment_profile.get());
+ cache_profile.update(&stats.file_cache_stats);
}
_tablet_reader->mutable_stats()->compressed_bytes_read = 0;
_tablet_reader->mutable_stats()->uncompressed_bytes_read = 0;
_tablet_reader->mutable_stats()->raw_rows_read = 0;
+ _tablet_reader->mutable_stats()->file_cache_stats = {};
- _bytes_read_from_local = _tablet_reader->stats().file_cache_stats.bytes_read_from_local;
- _bytes_read_from_remote = _tablet_reader->stats().file_cache_stats.bytes_read_from_remote;
+ _bytes_read_from_local = 0;
+ _bytes_read_from_remote = 0;
Review Comment:
The member variables `_bytes_read_from_local` and `_bytes_read_from_remote` are now dead code. Since `file_cache_stats` is reset to `{}` at the end of each cycle, these variables are always set to 0 at lines 718-719 and never read before being re-assigned. They should be removed from both the `.cpp` and `.h` files to avoid confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4301553777
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4302892139
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (11/11)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.53% (27513/37418) |
| Line Coverage | 57.31% (297581/519225) |
| Region Coverage | 54.48% (247344/454029) |
| Branch Coverage | 56.07% (107129/191070) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4010105659
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4003514699
<details>
<summary>TPC-DS: <b>Total hot run time: 153118 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f9f189378b0b0963c69a19d0697dab1f296dee0, data reload: false
query5 4326 667 520 520
query6 327 228 204 204
query7 4210 487 267 267
query8 328 239 225 225
query9 8718 2751 2732 2732
query10 523 412 331 331
query11 7338 5877 5533 5533
query12 179 136 125 125
query13 1261 470 373 373
query14 5778 3833 3538 3538
query14_1 2815 2828 2828 2828
query15 204 197 176 176
query16 1014 459 422 422
query17 1073 697 597 597
query18 2432 442 342 342
query19 214 206 177 177
query20 140 129 128 128
query21 225 148 117 117
query22 4904 5058 5020 5020
query23 16635 16059 15633 15633
query23_1 15875 15873 16070 15873
query24 7534 1727 1285 1285
query24_1 1280 1261 1275 1261
query25 664 539 496 496
query26 1280 298 176 176
query27 2769 499 289 289
query28 4480 1897 1875 1875
query29 873 607 495 495
query30 311 253 213 213
query31 1371 1319 1220 1220
query32 85 76 72 72
query33 518 354 289 289
query34 927 981 560 560
query35 662 698 614 614
query36 1105 1156 951 951
query37 147 97 86 86
query38 2949 2941 2841 2841
query39 891 867 851 851
query39_1 824 881 834 834
query40 241 163 144 144
query41 67 66 69 66
query42 319 311 305 305
query43 247 250 225 225
query44
query45 196 187 181 181
query46 920 1026 614 614
query47 2132 2138 2059 2059
query48 340 337 233 233
query49 633 481 391 391
query50 712 285 214 214
query51 4156 4141 4071 4071
query52 286 295 292 292
query53 295 351 286 286
query54 312 287 280 280
query55 91 88 87 87
query56 332 326 315 315
query57 1363 1319 1293 1293
query58 303 286 280 280
query59 1313 1431 1294 1294
query60 353 337 317 317
query61 144 138 146 138
query62 632 590 543 543
query63 322 281 276 276
query64 5068 1277 980 980
query65
query66 1443 493 353 353
query67 16230 16423 16313 16313
query68
query69 394 309 332 309
query70 975 923 942 923
query71 347 307 288 288
query72 2781 2735 2432 2432
query73 557 585 327 327
query74 9990 9901 9781 9781
query75 2948 2776 2458 2458
query76 2281 1133 696 696
query77 384 407 301 301
query78 11209 11281 10704 10704
query79 3134 860 603 603
query80 1731 642 532 532
query81 594 279 243 243
query82 1019 150 117 117
query83 324 265 245 245
query84 254 118 96 96
query85 904 485 427 427
query86 453 336 295 295
query87 3123 3160 2986 2986
query88 3588 2636 2639 2636
query89 457 374 340 340
query90 1985 183 171 171
query91 162 156 133 133
query92 78 72 76 72
query93 1387 844 511 511
query94 648 313 282 282
query95 581 408 312 312
query96 657 543 227 227
query97 2550 2554 2436 2436
query98 234 237 217 217
query99 1031 987 918 918
Total cold run time: 237453 ms
Total hot run time: 153118 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4600435659
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4088803234
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4094879559
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4004497073
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (11/11)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61072_9f9f189378b0b0963c69a19d0697dab1f296dee0_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61072_9f9f189378b0b0963c69a19d0697dab1f296dee0_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.49% (26166/36603) |
| Line Coverage | 54.25% (274195/505438) |
| Region Coverage | 51.58% (228089/442241) |
| Branch Coverage | 52.88% (97945/185226) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4301773625
# BE UT Coverage Report
Increment line coverage `0.00% (0/11)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/1b2a55f10b8075c06dcf0cfda108d38d4028ad7d_1b2a55f10b8075c06dcf0cfda108d38d4028ad7d/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.33% (20367/38193) |
| Line Coverage | 36.87% (191898/520520) |
| Region Coverage | 33.20% (149323/449703) |
| Branch Coverage | 34.29% (65266/190340) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4302288067
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #61072:
URL: https://github.com/apache/doris/pull/61072#discussion_r3338538019
##########
be/src/exec/scan/olap_scanner.cpp:
##########
@@ -681,25 +681,30 @@ void OlapScanner::update_realtime_counters() {
stats.compressed_bytes_read);
} else {
_state->get_query_ctx()->resource_ctx()->io_context()->update_scan_bytes_from_local_storage(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
_state->get_query_ctx()
->resource_ctx()
->io_context()
->update_scan_bytes_from_remote_storage(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
DorisMetrics::instance()->query_scan_bytes_from_local->increment(
- stats.file_cache_stats.bytes_read_from_local - _bytes_read_from_local);
+ stats.file_cache_stats.bytes_read_from_local);
DorisMetrics::instance()->query_scan_bytes_from_remote->increment(
- stats.file_cache_stats.bytes_read_from_remote - _bytes_read_from_remote);
+ stats.file_cache_stats.bytes_read_from_remote);
+ }
+
+ if (config::is_cloud_mode() && config::enable_file_cache) {
Review Comment:
done fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4602074532
<details>
<summary>TPC-H: <b>Total hot run time: 29727 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7032373e9a616e61893b6a8b067deb91b26a267b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17742 4184 4101 4101
q2 q3 10747 1427 831 831
q4 4688 487 348 348
q5 7538 911 601 601
q6 191 175 137 137
q7 785 844 663 663
q8 9365 1683 1653 1653
q9 5905 4491 4542 4491
q10 6785 1854 1546 1546
q11 441 280 258 258
q12 642 424 289 289
q13 18173 3537 2826 2826
q14 270 259 243 243
q15 q16 821 879 715 715
q17 971 963 921 921
q18 7238 5784 5652 5652
q19 1331 1329 1068 1068
q20 518 401 261 261
q21 6183 2843 2799 2799
q22 461 389 324 324
Total cold run time: 100795 ms
Total hot run time: 29727 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5168 4891 4893 4891
q2 q3 4932 5355 4661 4661
q4 2150 2219 1430 1430
q5 4938 4958 4758 4758
q6 240 177 128 128
q7 1844 1790 1634 1634
q8 2523 2259 2194 2194
q9 8080 7854 7482 7482
q10 4767 4676 4196 4196
q11 547 382 352 352
q12 745 736 525 525
q13 3070 3394 2846 2846
q14 268 285 252 252
q15 q16 680 697 613 613
q17 1305 1277 1259 1259
q18 7388 6812 6761 6761
q19 1106 1089 1123 1089
q20 2227 2220 1969 1969
q21 5367 4673 4512 4512
q22 535 456 415 415
Total cold run time: 57880 ms
Total hot run time: 51967 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](filecache) avoid duplicated FileCache counter accumulation in NewOlapScanner [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61072:
URL: https://github.com/apache/doris/pull/61072#issuecomment-4602155463
<details>
<summary>TPC-DS: <b>Total hot run time: 168982 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7032373e9a616e61893b6a8b067deb91b26a267b, data reload: false
query5 4332 631 495 495
query6 448 199 183 183
query7 4804 564 305 305
query8 373 224 215 215
query9 8754 3984 3983 3983
query10 445 316 260 260
query11 5926 2392 2165 2165
query12 153 100 96 96
query13 1272 618 419 419
query14 6451 5406 5084 5084
query14_1 4430 4420 4386 4386
query15 204 200 175 175
query16 1007 446 437 437
query17 1110 690 563 563
query18 2455 481 347 347
query19 196 183 143 143
query20 110 106 109 106
query21 218 140 119 119
query22 13679 13506 13287 13287
query23 17507 16532 16204 16204
query23_1 16265 16187 16335 16187
query24 7754 1779 1324 1324
query24_1 1316 1311 1313 1311
query25 547 450 375 375
query26 1294 320 168 168
query27 2695 542 343 343
query28 4465 2031 1984 1984
query29 1087 610 500 500
query30 312 235 195 195
query31 1118 1072 939 939
query32 103 60 61 60
query33 529 318 261 261
query34 1182 1141 652 652
query35 776 794 661 661
query36 1424 1378 1235 1235
query37 155 104 94 94
query38 3168 3088 3057 3057
query39 937 913 892 892
query39_1 871 880 877 877
query40 230 120 99 99
query41 64 61 63 61
query42 96 92 94 92
query43 318 324 278 278
query44
query45 196 186 178 178
query46 1108 1196 754 754
query47 2357 2395 2265 2265
query48 407 413 302 302
query49 658 469 346 346
query50 969 356 258 258
query51 4283 4305 4234 4234
query52 87 90 76 76
query53 256 273 197 197
query54 274 219 192 192
query55 77 78 69 69
query56 235 227 229 227
query57 1446 1398 1301 1301
query58 244 215 228 215
query59 1585 1669 1460 1460
query60 283 263 225 225
query61 164 158 164 158
query62 698 646 575 575
query63 236 183 187 183
query64 2642 875 689 689
query65
query66 1846 501 363 363
query67 29697 29603 29573 29573
query68
query69 431 314 277 277
query70 965 995 953 953
query71 294 225 205 205
query72 3015 2695 2397 2397
query73 869 788 447 447
query74 5134 4967 4791 4791
query75 2652 2572 2246 2246
query76 2299 1153 785 785
query77 348 389 286 286
query78 12347 12417 11860 11860
query79 1335 1087 767 767
query80 525 467 380 380
query81 446 287 234 234
query82 232 158 130 130
query83 298 275 249 249
query84 263 145 115 115
query85 848 530 445 445
query86 321 302 278 278
query87 3339 3346 3208 3208
query88 3617 2732 2731 2731
query89 416 385 328 328
query90 2173 190 182 182
query91 180 170 135 135
query92 63 57 58 57
query93 1445 1556 849 849
query94 522 362 315 315
query95 684 466 361 361
query96 1053 794 328 328
query97 2712 2699 2592 2592
query98 212 209 205 205
query99 1141 1174 1040 1040
Total cold run time: 251035 ms
Total hot run time: 168982 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org