You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/08/12 09:52:09 UTC
[GitHub] [doris] xinyiZzz opened a new pull request, #11740: [enhancement](memtracker) Optimize query memory accuracy
xinyiZzz opened a new pull request, #11740:
URL: https://github.com/apache/doris/pull/11740
# Proposed changes
Issue Number: close #11738
## Problem summary
### motivation
The value of the query mem tracker is consistent with the physical memory actually used by the query.
### problem causes
Currently, only the virtual memory used by the query can be tracked through the tcmalloc hook. When the memory is not fully used after the application, the recorded virtual memory will be larger than the physical memory.
At present, it is mainly because PODArray does not memset 0 when applying for memory, and blocks applied for through PODArray in places such as VOlapScanNode::_free_blocks are usually used for memory reuse and cannot be fully used.
### Fix
The query mem tracker only records the peak memory used by PODArray and MemPool
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #11740: [enhancement](memtracker) Optimize query memory accuracy
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #11740:
URL: https://github.com/apache/doris/pull/11740#discussion_r944670632
##########
be/src/runtime/memory/chunk_allocator.cpp:
##########
@@ -147,7 +147,7 @@ ChunkAllocator::ChunkAllocator(size_t reserve_limit)
}
Status ChunkAllocator::allocate(size_t size, Chunk* chunk) {
- DCHECK(BitUtil::RoundUpToPowerOfTwo(size) == size);
+ DCHECK((size & (size - 1)) == 0);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #11740: [enhancement](memtracker) Optimize query memory accuracy
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #11740:
URL: https://github.com/apache/doris/pull/11740#discussion_r944676056
##########
be/src/vec/common/pod_array.h:
##########
@@ -111,15 +112,26 @@ class PODArrayBase : private boost::noncopyable,
return byte_size(num_elements) + pad_right + pad_left;
}
+ inline void reset_peak() {
+ if (c_end - c_end_peak > 1024) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #11740: [enhancement](memtracker) Optimize query memory accuracy
Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #11740:
URL: https://github.com/apache/doris/pull/11740
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11740: [enhancement](memtracker) Optimize query memory accuracy
Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #11740:
URL: https://github.com/apache/doris/pull/11740#discussion_r944376360
##########
be/src/runtime/memory/chunk_allocator.cpp:
##########
@@ -147,7 +147,7 @@ ChunkAllocator::ChunkAllocator(size_t reserve_limit)
}
Status ChunkAllocator::allocate(size_t size, Chunk* chunk) {
- DCHECK(BitUtil::RoundUpToPowerOfTwo(size) == size);
+ DCHECK((size & (size - 1)) == 0);
Review Comment:
CHECK here, not use DCHECK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #11740: [enhancement](memtracker) Optimize query memory accuracy
Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #11740:
URL: https://github.com/apache/doris/pull/11740#discussion_r944312491
##########
be/src/vec/common/pod_array.h:
##########
@@ -111,15 +112,26 @@ class PODArrayBase : private boost::noncopyable,
return byte_size(num_elements) + pad_right + pad_left;
}
+ inline void reset_peak() {
+ if (c_end - c_end_peak > 1024) {
Review Comment:
add unlikely here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org