You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/12/31 21:15:12 UTC
[GitHub] [druid] jasonk000 commented on issue #11414: Kafka ingestion lag spikes up whenever tasks are rolling
jasonk000 commented on issue #11414:
URL: https://github.com/apache/druid/issues/11414#issuecomment-1003452368
@harinirajendran I recommend you experiment running the overlord with these three PRs pulled to your code: #12096 #12097 #12099, and let us know how it goes.
I reviewed your analysis, and some of the code, and took a profile on our cluster here.
You are correct that during task rollover the overlord gets busy processing `RunNotice` notices. I can identify two codepaths where RunNotice hits the TaskQueue (in purple):
On our system, with above fixes, TaskQueue is only a fraction of the time
- `SeekableStreamSupervisor.RunNotice::handle -> SeekableStreamSupervisor::runInternal -> SeekableStreamSupervisor::createNewTasks -> SeekableStreamSupervisor::createTasksForGroup -> TaskQueue::add`
- `SeekableStreamSupervisor.RunNotice::handle -> SeekableStreamSupervisor::runInternal -> SeekableStreamSupervisor::checkPendingCompletionTasks -> SeekableStreamSupervisor::killTasksInGroup -> SeekableStreamSupervisor::killTask -> TaskQueue::shutdown`
Both of these paths hit the lock in TaskQueue; the fixes I present above have improved scalability of TaskQueue on our system.
It might also help if you can share which metadata task storage engine you are using (SQL vs heap).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org