[GitHub] [incubator-doris] morningman opened a new issue #1995: Limit the memory consumption of Broker Scan Node

[GitBox <gi...@apache.org>]

morningman opened a new issue #1995: Limit the memory consumption of Broker Scan Node
URL: https://github.com/apache/incubator-doris/issues/1995
 
 
   A broker scan node will start one or more broker scanner thread to read the data,
   generate row batches and push them to row batch queue. And the upper caller will get
   row batch from this queue.
   
   Currently, we only limit the number of batches in queue, which is 1024. And if a row is very
   large, eg, 4K, and there are 1024 rows in a batch, so the max memory consumption of
   a batch queue is 4GB(4K * 1024 * 1024). And if there are many broker scan node on one
   Backend, it may lead to system OOM.
   
   So this memory consumption need to be limited.
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org