[GitHub] [incubator-doris] xinyiZzz opened a new issue #7553: [Proposal] Memory management optimization

[GitBox <gi...@apache.org>]

xinyiZzz opened a new issue #7553:
URL: https://github.com/apache/incubator-doris/issues/7553


   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Description
   
   https://shimo.im/docs/DXvQPcqPH3TyR6pY
   
   ## Motivation
   1. BE lacks memory access control, OOM frequently occurs during task execution,
   2. Users often report that BE consumes a lot of memory and cannot effectively locate the memory location;
   3. High concurrent query performance is poor, and the memory (cache) dynamically requested is low in multi-threaded throughput;
   4. There is a memory fragmentation problem, and the memory utilization rate needs to be further improved;
   5. Cannot effectively limit the memory usage of a single query;
   
   ## Expected Return
   stability:
   1. Introduce memory access control to reduce process crashes caused by insufficient memory, and reject task submission when memory is insufficient;
   2. The current memory usage statistics can be more accurately observed and printed at any time, and Top Query and memory hot spots can be located;
   
   performance:
   1. Improve the throughput of dynamic memory allocation and reduce direct application or release of memory from the system;
   2. Reduce memory fragmentation, memory reuse, and improve utilization;
   3. Reduce the additional overhead of auxiliary functions such as memory statistics;
   
   ### Use case
   
   _No response_
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org