You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "nsivarajan (via GitHub)" <gi...@apache.org> on 2026/04/23 03:49:12 UTC
[PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
nsivarajan opened a new pull request, #62739:
URL: https://github.com/apache/doris/pull/62739
### What problem does this PR solve?
Issue Number: close #62738
Related PR: #xxx
Problem Summary:
When `enable_memory_orphan_check=true` is set in be.conf, the Backend crashes
with the following assertion when cache background threads are triggered under
load:
Check failed: limiter_mem_tracker()->label() != "Orphan"
The affected threads are `cache_adjust_capacity_thread` and
`cache_prune_stale_thread` created in `daemon.cpp`. These threads perform
memory allocations without a `MemTrackerLimiter` attached, so they fall back
to the Orphan tracker. The crash is non-deterministic — it does not happen on
startup but fires when cache pressure activates these background threads
(e.g. cache near capacity or stale blocks present).
##### Root Cause
Both cache daemon threads are created without calling `SCOPED_ATTACH_TASK`.
Any thread without an explicit `MemTrackerLimiter` inherits the Orphan tracker.
When `enable_memory_orphan_check=true`, memory allocation on the Orphan tracker
triggers the assertion and crashes the BE.
##### Fix
Attach a dedicated `MemTrackerLimiter` to each cache daemon thread via
`SCOPED_ATTACH_TASK` before the work loop begins, consistent with how other
daemon threads in the codebase handle memory tracking.
##### Files Changed
`be/src/common/daemon.cpp` — add `SCOPED_ATTACH_TASK` to
`cache_adjust_capacity_thread` and `cache_prune_stale_thread`
##### Tests
- Verified no crash with `enable_memory_orphan_check=true` and file cache
enabled under TPC-DS query load
- Both cache threads now have a valid memory tracking context
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [X] No need to test or manual test. Explain why:
- [X] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [X] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [X] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #62739:
URL: https://github.com/apache/doris/pull/62739#issuecomment-4301610016
/run build all
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62739:
URL: https://github.com/apache/doris/pull/62739#issuecomment-4317419559
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62739:
URL: https://github.com/apache/doris/pull/62739#issuecomment-4315891648
# BE Regression && UT Coverage Report
Increment line coverage `33.33% (4/12)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62739_4e83c99036fb96d051a91fdd79aa4cd183477d05_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62739_4e83c99036fb96d051a91fdd79aa4cd183477d05_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.47% (26799/37497) |
| Line Coverage | 53.82% (279810/519870) |
| Region Coverage | 47.18% (215126/455969) |
| Branch Coverage | 50.49% (97376/192877) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #62739:
URL: https://github.com/apache/doris/pull/62739#issuecomment-4302119586
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (6/6)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/62739_8571a35e1390d524e797604ac8f44df713bc2968_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/62739_8571a35e1390d524e797604ac8f44df713bc2968_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 73.61% (27544/37418) |
| Line Coverage | 57.39% (298004/519226) |
| Region Coverage | 54.56% (247733/454027) |
| Branch Coverage | 56.18% (107349/191066) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #62739:
URL: https://github.com/apache/doris/pull/62739#discussion_r3138497880
##########
be/src/common/daemon.cpp:
##########
@@ -507,6 +508,12 @@ void Daemon::je_reset_dirty_decay_thread() const {
}
void Daemon::cache_adjust_capacity_thread() {
+ // Attach a tracker so Block/Column destructors triggered by LRU eviction
+ // do not hit the memory_orphan_check DCHECK.
+ auto mem_tracker = MemTrackerLimiter::create_shared(MemTrackerLimiter::Type::GLOBAL,
Review Comment:
The root cause is that the cache value in the query cache contains a block. When the daemon performs pruning, it releases the cache value, which in turn releases the block, resulting in a dcheck crash.
Your current approach can prevent the dcheck from crashing, but it will cause incorrect statistics in the memtracker of the query cache.
In my opinion, a more reasonable solution is to override the adjust_capacity_weighted method in the query cache. First, add SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER(ExecEnv::GetInstance()->query_cache_mem_tracker());, then continue to call the parent class's adjust_capacity_weighted. This ensures that if any destructor runs during the execution of this adjust_capacity_weighted function, the memory will be deducted from the query cache's memtracker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62739:
URL: https://github.com/apache/doris/pull/62739#issuecomment-4317419461
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #62739:
URL: https://github.com/apache/doris/pull/62739#issuecomment-4317419585
skip check_coverage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on PR #62739:
URL: https://github.com/apache/doris/pull/62739#issuecomment-4315004742
/run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #62739:
URL: https://github.com/apache/doris/pull/62739#issuecomment-4301604591
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "nsivarajan (via GitHub)" <gi...@apache.org>.
nsivarajan commented on code in PR #62739:
URL: https://github.com/apache/doris/pull/62739#discussion_r3139266634
##########
be/src/common/daemon.cpp:
##########
@@ -507,6 +508,12 @@ void Daemon::je_reset_dirty_decay_thread() const {
}
void Daemon::cache_adjust_capacity_thread() {
+ // Attach a tracker so Block/Column destructors triggered by LRU eviction
+ // do not hit the memory_orphan_check DCHECK.
+ auto mem_tracker = MemTrackerLimiter::create_shared(MemTrackerLimiter::Type::GLOBAL,
Review Comment:
Thanks for the analysis. You're right — my original fix attached a generic tracker to the daemon thread(fixing symptom) , which prevented the DCHECK but caused Block memory to be deducted from the wrong tracker instead of query_cache_mem_tracker.
updated with suggested correct fix, to override adjust_capacity_weighted, reset_initial_capacity and prune_stale in QueryCache with SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER, so Block destructors during any eviction path always charge the right tracker. Reverted the daemon.cpp change accordingly.
Please take another look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) attach MemTracker to cache background threads to fix orphan crash [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #62739:
URL: https://github.com/apache/doris/pull/62739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org