[GitHub] [incubator-doris] yangzhg opened a new issue #6978: [Feature] Limit Doris's memory usage more accurately and reduce the occurrence of OOM

[GitBox <gi...@apache.org>]

yangzhg opened a new issue #6978:
URL: https://github.com/apache/incubator-doris/issues/6978


   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Description
   
   Currently, the main reason that causes doris to exit abnormally is that it occupies too much memory and OOM is killed by the system. Although doris has a mem_limit configuration, the memory usage limit is not strict. As a result, some memory is not recorded and consumed in doris. There are two main parts of the rough:
   1. The memory that can be tracked by memtracker is mainly occupied by various mempools
   2. Memory that cannot be tracked through memtracker, mainly memory requested through new/malloc
   The current problem is as follows:
   1. The memory that can be tracked by memtracker does not consider the total memory occupied by the current system when consuming, but only considers the mem limit of the current tracker. This situation is particularly prominent on the memtracker without memory display.
   2. There are currently no restrictions on the memory requested through new/malloc
   
   ### Use case
   
   In order to solve the above two problems, what we need to do is as follows:
   1. For the memory that can be tracked through the mem tracker, when you consume, check whether the used memory exceeds the system memory limit. If it exceeds the system memory limit, the application will fail directly.
   2. For memory that cannot be tracked by the system, a hook for memory allocation needs to be set. Establish a mapping relationship between memory application and query. If the memory allocation exceeds the limit, the corresponding query will be cancelled.
   3. Organize internal allocation relationships to reduce unnecessary memory allocation
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org