You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jacktengg (via GitHub)" <gi...@apache.org> on 2026/04/28 10:33:38 UTC
[PR] [improve](memory) Improve BE memory profile accuracy for jemalloc internal memory, segment metadata memory, and BloomFilter memory tracking [doris]
jacktengg opened a new pull request, #62909:
URL: https://github.com/apache/doris/pull/62909
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
BE memory overview could overstate tracked memory and report misleading
jemalloc/segment memory details.
The main issues were:
1. Segment metadata accounting estimated primary-key BloomFilter bytes in `Segment::_open()`, even though BloomFilter buffers are now allocated
through Doris `Allocator` and tracked by MemTracker. This could double
count PK BloomFilter memory.
2. Segment metadata accounting estimated short-key index memory unconditionally, even when a segment footer has no short-key index page.
3. Jemalloc “cache” memory was calculated from `tcache + dirty pages`, which misses allocator resident overhead such as fragmentation/slack and
page rounding, and does not represent physical jemalloc overhead
accurately.
4. BloomFilter buffers used raw allocation paths, so large BloomFilter memory could bypass Doris MemTracker accounting.
This change improves memory accounting by:
- Computing jemalloc internal resident overhead from `stats.resident - stats.allocated`, while exposing detailed jemalloc counters including resident, allocated, active, tcache, metadata, dirty, and muzzy bytes.
- Removing PK BloomFilter bytes from Segment metadata estimation because the actual BloomFilter buffer is now allocator-tracked.
- Counting short-key index estimate only when the footer has a short-key index page.
- Refactoring BloomFilter buffer allocation to use Doris `Allocator`, and changing NGramBloomFilter storage to allocator-aware `DorisVector`.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](memory) Improve BE memory profile accuracy for jemalloc internal memory, segment metadata memory, and BloomFilter memory tracking [doris]
Posted by "github-actions[bot] (via GitHub) via commits" <co...@doris.apache.org>.
github-actions[bot] closed pull request #62909: [improve](memory) Improve BE memory profile accuracy for jemalloc internal memory, segment metadata memory, and BloomFilter memory tracking
URL: https://github.com/apache/doris/pull/62909
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](memory) Improve BE memory profile accuracy for jemalloc internal memory, segment metadata memory, and BloomFilter memory tracking [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #62909:
URL: https://github.com/apache/doris/pull/62909#issuecomment-4334461903
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](memory) Improve BE memory profile accuracy for jemalloc internal memory, segment metadata memory, and BloomFilter memory tracking [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #62909:
URL: https://github.com/apache/doris/pull/62909#issuecomment-6405426384
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org