[GitHub] [doris] xinyiZzz opened a new pull request, #10743: [enhancement] Refactor to improve the usability of MemTracker

[GitBox <gi...@apache.org>]

xinyiZzz opened a new pull request, #10743:
URL: https://github.com/apache/doris/pull/10743

   # Proposed changes
   
   Issue Number: close #10741
   
   ## Problem Summary:
   
   #### Usability issues
   The new MemTracker is difficult to use, hoping to reduce the cost of use, mainly:
   
   1. `SCOPED_SWITCH_xxx_TRACKER` related macros for switching TLS trackers are various and depend on each other, which is difficult to use;
     Most of the macros are mainly to improve performance and code robustness, because MemTracker currently uses std::shared_ptr in TLS to have performance problems under multi-threading, which is solved by a complex caching mechanism.
   
   2. Creating a new MemTracker is error-prone;
     Because the hierarchical relationship between MemTrackers needs to be considered to avoid statistical errors affecting the accuracy of the overall statistics;
   
   #### Solution
   1. Divide MemTracker into two subclasses,
       1) The first type of MemTrackerLimiter, including four layers of process, task pool, query/load task traker, and instance tracker, is used to track and limit the memory usage of processes and Query.
           - Manual consumption of Tracker is prohibited, only automatic consumption in TCMalloc Hook is allowed to ensure accuracy.
           - PageCache transfers memory ownership only when memory allocation and release occur.
       2) The second type of MemTrackerObserve, all operators and other MemTrackers, are only used to track the memory usage of the specified code segment,
           - There is no parent-child relationship between MemTrackerObserves. Both fathers are instance trakcers, but their consumption will not consume instance trakcers synchronously. Therefore, errors in statistics will not affect the memory tracking and restrictions of processes and Query;
           - When the Hook consumes the thread tls MemTracker, it will consume the MemTrackerLimiter and all MemTrackerObserves respectively.
   
   2. All MemTrackers use raw pointers to avoid performance problems caused by frequent switching of MemTrackers in tls;
   
   ## Checklist(Required)
   
   1. Does it affect the original behavior: (Yes/No/I Don't know)
   2. Has unit tests been added: (Yes/No/No Need)
   3. Has document been added or modified: (Yes/No/No Need)
   4. Does it need to update dependencies: (Yes/No)
   5. Are there any changes that cannot be rolled back: (Yes/No)
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org

[GitHub] [doris] yiguolei merged pull request #10743: [enhancement] Refactor to improve the usability of MemTracker

[GitBox <gi...@apache.org>]

yiguolei merged PR #10743:
URL: https://github.com/apache/doris/pull/10743


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org