You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/06/24 11:21:09 UTC
[GitHub] [doris] k-i-d-d opened a new pull request, #10409: [fix] (mem tracker) Fix inaccurate mem tracker leads to load OOM
k-i-d-d opened a new pull request, #10409:
URL: https://github.com/apache/doris/pull/10409
# Proposed changes
Issue Number: close #10407
## Problem Summary:
1. Fix `LoadChannelMgr mem tracker` statistic higher than actual value.
2. When the consumption of the mem tracker in the TCMalloc Hook exceeds the limit, the memory will be consumed repeatedly.
When ssb 100G is imported, the `LoadChannelMgr mem tracker` is prone to an excess limit. A large number of repeated consumptions cause the `Process mem tracker` to also exceed the limit, and eventually all queries and imports will fail.
Incomplete fix: The current modification disables the `LoadChannelMgr mem tracker`, which will cause the load channel memory to not be properly limited, will be fixed in the next pr.
## Checklist(Required)
1. Does it affect the original behavior: (Yes)
3. Has unit tests been added: (No Need)
4. Has document been added or modified: (No Need)
5. Does it need to update dependencies: (No)
6. Are there any changes that cannot be rolled back: (No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10409: [fix] (mem tracker) Fix inaccurate mem tracker leads to load OOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10409:
URL: https://github.com/apache/doris/pull/10409#issuecomment-1166201849
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] k-i-d-d commented on a diff in pull request #10409: [fix] (mem tracker) Fix inaccurate mem tracker leads to load OOM
Posted by GitBox <gi...@apache.org>.
k-i-d-d commented on code in PR #10409:
URL: https://github.com/apache/doris/pull/10409#discussion_r914338916
##########
be/src/runtime/load_channel.h:
##########
@@ -39,9 +39,8 @@ class Cache;
// corresponding to a certain load job
class LoadChannel {
public:
- LoadChannel(const UniqueId& load_id, int64_t load_mem_limit, int64_t channel_mem_limit,
- int64_t timeout_s, bool is_high_priority, const std::string& sender_ip,
- bool is_vec);
+ LoadChannel(const UniqueId& load_id, std::shared_ptr<MemTracker>& mem_tracker,
+ int64_t timeout_s, bool is_high_priority, const std::string& sender_ip, bool is_ve);
Review Comment:
should be `is_vec`, fix it in next pr, thks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #10409: [fix] (mem tracker) Fix inaccurate mem tracker leads to load OOM
Posted by GitBox <gi...@apache.org>.
dataroaring merged PR #10409:
URL: https://github.com/apache/doris/pull/10409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yangzhg commented on a diff in pull request #10409: [fix] (mem tracker) Fix inaccurate mem tracker leads to load OOM
Posted by GitBox <gi...@apache.org>.
yangzhg commented on code in PR #10409:
URL: https://github.com/apache/doris/pull/10409#discussion_r906928019
##########
be/src/runtime/load_channel.h:
##########
@@ -39,9 +39,8 @@ class Cache;
// corresponding to a certain load job
class LoadChannel {
public:
- LoadChannel(const UniqueId& load_id, int64_t load_mem_limit, int64_t channel_mem_limit,
- int64_t timeout_s, bool is_high_priority, const std::string& sender_ip,
- bool is_vec);
+ LoadChannel(const UniqueId& load_id, std::shared_ptr<MemTracker>& mem_tracker,
+ int64_t timeout_s, bool is_high_priority, const std::string& sender_ip, bool is_ve);
Review Comment:
why rename to is_ve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #10409: [fix] (mem tracker) Fix inaccurate mem tracker leads to load OOM
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #10409:
URL: https://github.com/apache/doris/pull/10409#issuecomment-1166201854
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] k-i-d-d commented on a diff in pull request #10409: [fix] (mem tracker) Fix inaccurate mem tracker leads to load OOM
Posted by GitBox <gi...@apache.org>.
k-i-d-d commented on code in PR #10409:
URL: https://github.com/apache/doris/pull/10409#discussion_r914338916
##########
be/src/runtime/load_channel.h:
##########
@@ -39,9 +39,8 @@ class Cache;
// corresponding to a certain load job
class LoadChannel {
public:
- LoadChannel(const UniqueId& load_id, int64_t load_mem_limit, int64_t channel_mem_limit,
- int64_t timeout_s, bool is_high_priority, const std::string& sender_ip,
- bool is_vec);
+ LoadChannel(const UniqueId& load_id, std::shared_ptr<MemTracker>& mem_tracker,
+ int64_t timeout_s, bool is_high_priority, const std::string& sender_ip, bool is_ve);
Review Comment:
should be `is_vec`, fix it in next pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org