You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by mandeep gandhi <we...@gmail.com> on 2020/03/16 17:57:51 UTC
Connect - Membership Protocol
Hi,
(PS - I did not use users list as this concerns some internals)
I was trying to deploy the following config on a K8s cluster -
Namespace - X1 Connect group id - G1 Bootstrap servers - B1, B2
Namespace - X2 Connect group id - G1 Bootstrap servers - B3, B4
With this configuration, I am seeing multiple rebalances (logs below). So
the question is how is the Membership protocol working?
I have tried to follow some KIPs and they mostly say that the Bootstrap
server is the coordinator of the group. If yes, then logically this
configuration should work just fine as both configs have different
bootstrap servers. But as far as I have tried to understand the code (and
tried to run Kafka Connect), it seems like the members get added in the
group one by one and the head of the list becomes the group co-ordinator. (
if I change Connect Group id in 2nd config, things work)
Also, I wanted some pointers on what is happening at the server-side vs
client-side during the protocol.
Kindly help.
Logs -
[2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Current config state offset 12 does not
match group assignment 9164. Forcing rebalance.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)
[2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Rebalance started
(org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:213)
[2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
groupId=streaming-connect] (Re-)joining group
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
[2020-03-16 10:27:13,388] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Successfully joined group with generation
18535 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:469)
[2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Joined group at generation 18535 and got
assignment: Assignment{error=0,
leader='connect-1-3580fdcc-257e-40e7-8243-f7839021599f',
leaderUrl='http://10.13.191.32:8083/', offset=9164, connectorIds=[],
taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0}
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:1397)
[2020-03-16 10:27:13,389] WARN [Worker clientId=connect-1,
groupId=streaming-connect] Catching up to assignment's config offset.
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:909)
[2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1,
groupId=streaming-connect] Current config state offset 12 is behind
group assignment 9164, reading to end of config log
(org.apache.kafka.connect.runtime.distributed.DistributedHerder:970)
Thanks,
Mandeep Gandhi
Re: Connect - Membership Protocol
Posted by Robin Moffatt <ro...@confluent.io>.
Are the four brokers across two clusters (B1+B2) / (B3+B4), or one cluster?
If one cluster, are using the same config/offset/status topics for each
Connect cluster? Because that definitely won't work.
In case it's useful:
https://rmoff.net/2019/11/22/common-mistakes-made-when-configuring-multiple-kafka-connect-workers/
--
Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
On Tue, 17 Mar 2020 at 17:58, mandeep gandhi <we...@gmail.com>
wrote:
> Gentle reminder.
>
> + users.
>
> On Mon, Mar 16, 2020 at 11:27 PM mandeep gandhi <we...@gmail.com>
> wrote:
>
> > Hi,
> >
> > (PS - I did not use users list as this concerns some internals)
> >
> > I was trying to deploy the following config on a K8s cluster -
> >
> > Namespace - X1 Connect group id - G1 Bootstrap servers - B1, B2
> > Namespace - X2 Connect group id - G1 Bootstrap servers - B3, B4
> >
> > With this configuration, I am seeing multiple rebalances (logs below). So
> > the question is how is the Membership protocol working?
> > I have tried to follow some KIPs and they mostly say that the Bootstrap
> > server is the coordinator of the group. If yes, then logically this
> > configuration should work just fine as both configs have different
> > bootstrap servers. But as far as I have tried to understand the code
> (and
> > tried to run Kafka Connect), it seems like the members get added in the
> > group one by one and the head of the list becomes the group
> co-ordinator. (
> > if I change Connect Group id in 2nd config, things work)
> >
> > Also, I wanted some pointers on what is happening at the server-side vs
> > client-side during the protocol.
> >
> >
> > Kindly help.
> >
> > Logs -
> >
> > [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Current config state offset 12 does not match
> group assignment 9164. Forcing rebalance.
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)
> > [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Rebalance started
> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:213)
> > [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] (Re-)joining group
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
> > [2020-03-16 10:27:13,388] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Successfully joined group with generation 18535
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:469)
> > [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Joined group at generation 18535 and got
> assignment: Assignment{error=0,
> leader='connect-1-3580fdcc-257e-40e7-8243-f7839021599f', leaderUrl='
> http://10.13.191.32:8083/', offset=9164, connectorIds=[], taskIds=[],
> revokedConnectorIds=[], revokedTaskIds=[], delay=0}
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1397)
> > [2020-03-16 10:27:13,389] WARN [Worker clientId=connect-1,
> groupId=streaming-connect] Catching up to assignment's config offset.
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:909)
> > [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Current config state offset 12 is behind group
> assignment 9164, reading to end of config log
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:970)
> >
> >
> >
> > Thanks,
> >
> > Mandeep Gandhi
> >
> >
> >
> >
>
Re: Connect - Membership Protocol
Posted by mandeep gandhi <we...@gmail.com>.
Gentle reminder.
+ users.
On Mon, Mar 16, 2020 at 11:27 PM mandeep gandhi <we...@gmail.com>
wrote:
> Hi,
>
> (PS - I did not use users list as this concerns some internals)
>
> I was trying to deploy the following config on a K8s cluster -
>
> Namespace - X1 Connect group id - G1 Bootstrap servers - B1, B2
> Namespace - X2 Connect group id - G1 Bootstrap servers - B3, B4
>
> With this configuration, I am seeing multiple rebalances (logs below). So
> the question is how is the Membership protocol working?
> I have tried to follow some KIPs and they mostly say that the Bootstrap
> server is the coordinator of the group. If yes, then logically this
> configuration should work just fine as both configs have different
> bootstrap servers. But as far as I have tried to understand the code (and
> tried to run Kafka Connect), it seems like the members get added in the
> group one by one and the head of the list becomes the group co-ordinator. (
> if I change Connect Group id in 2nd config, things work)
>
> Also, I wanted some pointers on what is happening at the server-side vs
> client-side during the protocol.
>
>
> Kindly help.
>
> Logs -
>
> [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, groupId=streaming-connect] Current config state offset 12 does not match group assignment 9164. Forcing rebalance. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)
> [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, groupId=streaming-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:213)
> [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, groupId=streaming-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
> [2020-03-16 10:27:13,388] INFO [Worker clientId=connect-1, groupId=streaming-connect] Successfully joined group with generation 18535 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:469)
> [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1, groupId=streaming-connect] Joined group at generation 18535 and got assignment: Assignment{error=0, leader='connect-1-3580fdcc-257e-40e7-8243-f7839021599f', leaderUrl='http://10.13.191.32:8083/', offset=9164, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1397)
> [2020-03-16 10:27:13,389] WARN [Worker clientId=connect-1, groupId=streaming-connect] Catching up to assignment's config offset. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:909)
> [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1, groupId=streaming-connect] Current config state offset 12 is behind group assignment 9164, reading to end of config log (org.apache.kafka.connect.runtime.distributed.DistributedHerder:970)
>
>
>
> Thanks,
>
> Mandeep Gandhi
>
>
>
>
Re: Connect - Membership Protocol
Posted by mandeep gandhi <we...@gmail.com>.
Gentle reminder.
+ users.
On Mon, Mar 16, 2020 at 11:27 PM mandeep gandhi <we...@gmail.com>
wrote:
> Hi,
>
> (PS - I did not use users list as this concerns some internals)
>
> I was trying to deploy the following config on a K8s cluster -
>
> Namespace - X1 Connect group id - G1 Bootstrap servers - B1, B2
> Namespace - X2 Connect group id - G1 Bootstrap servers - B3, B4
>
> With this configuration, I am seeing multiple rebalances (logs below). So
> the question is how is the Membership protocol working?
> I have tried to follow some KIPs and they mostly say that the Bootstrap
> server is the coordinator of the group. If yes, then logically this
> configuration should work just fine as both configs have different
> bootstrap servers. But as far as I have tried to understand the code (and
> tried to run Kafka Connect), it seems like the members get added in the
> group one by one and the head of the list becomes the group co-ordinator. (
> if I change Connect Group id in 2nd config, things work)
>
> Also, I wanted some pointers on what is happening at the server-side vs
> client-side during the protocol.
>
>
> Kindly help.
>
> Logs -
>
> [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, groupId=streaming-connect] Current config state offset 12 does not match group assignment 9164. Forcing rebalance. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)
> [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, groupId=streaming-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:213)
> [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, groupId=streaming-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
> [2020-03-16 10:27:13,388] INFO [Worker clientId=connect-1, groupId=streaming-connect] Successfully joined group with generation 18535 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:469)
> [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1, groupId=streaming-connect] Joined group at generation 18535 and got assignment: Assignment{error=0, leader='connect-1-3580fdcc-257e-40e7-8243-f7839021599f', leaderUrl='http://10.13.191.32:8083/', offset=9164, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1397)
> [2020-03-16 10:27:13,389] WARN [Worker clientId=connect-1, groupId=streaming-connect] Catching up to assignment's config offset. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:909)
> [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1, groupId=streaming-connect] Current config state offset 12 is behind group assignment 9164, reading to end of config log (org.apache.kafka.connect.runtime.distributed.DistributedHerder:970)
>
>
>
> Thanks,
>
> Mandeep Gandhi
>
>
>
>