Scheduler is taking a long time, but % and absolute value

[Doug Whitfield <dw...@perforce.com>]

Hi folks,

I would like some help understanding what is going on in the below:

[2021-05-03 12:53:26,856] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-42 in 1659 milliseconds, of which 1647 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)

[2021-05-03 12:53:26,871] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-43 in 1674 milliseconds, of which 1660 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)
[2021-05-03 12:53:26,871] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-13 in 1674 milliseconds, of which 1674 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager)

It seems from looking online that this is a long time for these to take place. What is odd to me is that it does not look like a timeout. Sometimes the scheduler takes the full time, and sometimes it is not the full time. The actual loading increases in time too.

I have been able to figure out where this log message was added to the code, but it doesn’t do a lot to help me understand the issue: https://github.com/apache/kafka/pull/8155

Ultimately, I don’t really know what the scheduler is doing (other than scheduling, presumably). Is Zookeeper involved in scheduling?

Aside from a virus scan about 12 hours before these logs, there is not any obvious pressure on the system, though this is not 100% clear either. We are working to make it more clear that the system is not under pressure.

I am definitely more of a systems person than a Kafka person, so there may be something obvious here I am missing on the Kafka side. I will not be offended!

Best Regards,
Doug


This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and any attachments and notify us immediately.