[GitHub] [incubator-doris] morningman opened a new pull request #1954: Limit the memory usage of Loading process

[GitBox <gi...@apache.org>]

morningman opened a new pull request #1954: Limit the memory usage of Loading process
URL: https://github.com/apache/incubator-doris/pull/1954
 
 
   Current load framework using memtable to receive incoming loading data and flush to disk when reaching limit (default is 100MB).
   Each tablet corresponds to a memtable, so that if there are many tablets on a Backend, and the loading data is distributed evenly on each of these tablets, the total memory consumption can be very large because all memtables will no be flushed until they reaching the 100MB.
   For example, if there are 100 tablets on a Backend, the peak memory consumption can be 10GB(100 * 100MB), and this may cause process killed by system OOM.
   
   This CL try to resolve this problem.
   
   ISSUE #1951

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org