[GitHub] [incubator-doris] morningman commented on issue #2065: Routine Load may takes too many FE memory

[GitBox <gi...@apache.org>]

morningman commented on issue #2065: Routine Load may takes too many FE memory
URL: https://github.com/apache/incubator-doris/issues/2065#issuecomment-547306633
 
 
   The reason is that `beIdToBatchTask` in RoutineLoadTaskScheduler is not cleaned. And more and more `TRoutineLoadTask` objects are added to this map.
   
   It is not cleaned because the `submitBatchTasksIfNotEmpty()` is not called because the following conditions are not met:
   
   ```
   if (needScheduleTasksQueue.size() == 0 || idleSlotNum == 0) {
       submitBatchTasksIfNotEmpty();
   }
   ```
   
   This can happen when we set FE config `max_running_txn_num_per_db` smaller than 
   (num of BE * `max_concurrent_task_num_per_be`). 
   Routine load tasks always failed to begin transaction due to exceed `max_running_txn_num_per_db` limit and be pushed back to the `needScheduleTasksQueue`. 
   
   So `needScheduleTasksQueue` is always non-empty, but there are idle slots available.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org