[GitHub] [druid] harinirajendran commented on issue #11414: Kafka ingestion lag spikes up whenever tasks are rolling

[GitBox <gi...@apache.org>]

harinirajendran commented on issue #11414:
URL: https://github.com/apache/druid/issues/11414#issuecomment-1030512254


   > I'm not sure whether the proposed PRs would fix your issue. I'd have to see stack traces during the 1.5-2mins pause time you refer, is that how you determined they were waiting on the future?
   
   I determined it is the task bootstrap time that's causing `run_notice` to take ~8s by reading through the run_notice handle code and also correlating the timestamps in the logs. 
   
   <img width="1467" alt="Screen Shot 2022-02-04 at 9 18 51 PM" src="https://user-images.githubusercontent.com/9054348/152626992-926dd6be-5cc8-4b20-a3f8-77c14460827e.png">
   
   For example, here you can see that the kafkasupervisor_<datasource> thread was stuck for 13 seconds without logging anything. So I went through the code path between these 2 log statements and that's where I hit the futures. And I found logs from other async threads in that gap trying to get `task_status` but were stuck in a retry loop as the tasks were bootstrapping and hence wasn't accepting connections for 5-6s.
   
   To prove this theory, I switched one of our lab environments to use indexers instead of MMs for real-time ingestion tasks and as I doubted, the `run_notice` handle time fell down from 8s to under 2s as there are no jvm restarts involved for every task bootstrap with indexers.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org