You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Upesh Desai <ud...@itrsgroup.com> on 2021/04/22 21:10:04 UTC
Kafka Connect Dist. Worker Does not join group
Hello,
Has anyone experienced this scenario before when having a distributed connect cluster:
[kafka-coordinator-heartbeat-thread | connect-cluster] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Worker clientId=connect-1, groupId=connect-cluster] Member
connect-1-50ee335c-6a0c-42d6-b3fc-0dc43067de0b sending LeaveGroup request to coordinator kafka-1:9092 (id: 2147483646 rack: null) due to consumer poll timeout has expired. This mean
s the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this
either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
I have left max.poll.interval.ms at the default of 5 minutes, and changed rebalance.timeout.ms=300000, yet still see this error occurring. This was with 2 connect worker instances, deployed on Kubernetes, and we see this error on a fresh install (no previous kafka data), and also see it when re-deploying SOME of the time, it is not consistent behavior.
Is there some other configuration I am missing?
Thanks in advance,
Upesh
Upesh Desai
Senior Software Developer
udesai@itrsgroup.com
www.itrsgroup.com
Internet communications are not secure and therefore the ITRS Group does not accept legal responsibility for the contents of this message. Any view or opinions presented are solely those of the author and do not necessarily represent those of the ITRS Group unless otherwise specifically stated.
[itrs.email.signature]
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
Re: Kafka Connect Dist. Worker Does not join group
Posted by Upesh Desai <ud...@itrsgroup.com>.
Hello, I wanted to follow up to see if anyone has ever experienced such behavior with Kafka Connect? Any guidance would be greatly appreciated!
Thanks,
Upesh
Upesh Desai | Senior Software Developer | udesai@itrsgroup.com
www.itrsgroup.com
From: Upesh Desai <ud...@itrsgroup.com>
Date: Thursday, April 22, 2021 at 4:10 PM
To: Users <us...@kafka.apache.org>
Subject: Kafka Connect Dist. Worker Does not join group
Hello,
Has anyone experienced this scenario before when having a distributed connect cluster:
[kafka-coordinator-heartbeat-thread | connect-cluster] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - [Worker clientId=connect-1, groupId=connect-cluster] Member
connect-1-50ee335c-6a0c-42d6-b3fc-0dc43067de0b sending LeaveGroup request to coordinator kafka-1:9092 (id: 2147483646 rack: null) due to consumer poll timeout has expired. This mean
s the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this
either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
I have left max.poll.interval.ms at the default of 5 minutes, and changed rebalance.timeout.ms=300000, yet still see this error occurring. This was with 2 connect worker instances, deployed on Kubernetes, and we see this error on a fresh install (no previous kafka data), and also see it when re-deploying SOME of the time, it is not consistent behavior.
Is there some other configuration I am missing?
Thanks in advance,
Upesh
[cid:image001.png@01D740DE.44D9A2B0]<https://www.itrsgroup.com/>
Upesh Desai
Senior Software Developer
udesai@itrsgroup.com<ma...@itrsgroup.com>
www.itrsgroup.com<https://www.itrsgroup.com/>
Internet communications are not secure and therefore the ITRS Group does not accept legal responsibility for the contents of this message. Any view or opinions presented are solely those of the author and do not necessarily represent those of the ITRS Group unless otherwise specifically stated.
[itrs.email.signature]
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.