[GitHub] [druid] jihoonson commented on issue #11414: Kafka ingestion lag spikes up whenever tasks are rolling

[GitBox <gi...@apache.org>]

jihoonson commented on issue #11414:
URL: https://github.com/apache/druid/issues/11414#issuecomment-875125389


   Hi @harinirajendran, thank you for your report. Was it only one task being paused or all tasks for the same datasource? If the first is the case, then maybe your guess is correct that there were lots of notices in the queue. If the second is the case, it means the supervisor was already in processing of the checkpoint notice but hadn't finished it yet. It was probably waiting for all tasks to respond to the pause request or waiting for tasks to respond to the setEndOffset request (see `SeekableStreamSupervisor.checkpointTaskGroup()`. One of our customers had a similar issue at Imply before and the cause was frequent HTTP communication failures which were retried with backoff.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org