You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by mandeep gandhi <we...@gmail.com> on 2020/03/17 17:57:54 UTC
Re: Connect - Membership Protocol
Gentle reminder.
+ users.
On Mon, Mar 16, 2020 at 11:27 PM mandeep gandhi <we...@gmail.com>
wrote:
> Hi,
>
> (PS - I did not use users list as this concerns some internals)
>
> I was trying to deploy the following config on a K8s cluster -
>
> Namespace - X1 Connect group id - G1 Bootstrap servers - B1, B2
> Namespace - X2 Connect group id - G1 Bootstrap servers - B3, B4
>
> With this configuration, I am seeing multiple rebalances (logs below). So
> the question is how is the Membership protocol working?
> I have tried to follow some KIPs and they mostly say that the Bootstrap
> server is the coordinator of the group. If yes, then logically this
> configuration should work just fine as both configs have different
> bootstrap servers. But as far as I have tried to understand the code (and
> tried to run Kafka Connect), it seems like the members get added in the
> group one by one and the head of the list becomes the group co-ordinator. (
> if I change Connect Group id in 2nd config, things work)
>
> Also, I wanted some pointers on what is happening at the server-side vs
> client-side during the protocol.
>
>
> Kindly help.
>
> Logs -
>
> [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, groupId=streaming-connect] Current config state offset 12 does not match group assignment 9164. Forcing rebalance. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)
> [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, groupId=streaming-connect] Rebalance started (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:213)
> [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1, groupId=streaming-connect] (Re-)joining group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
> [2020-03-16 10:27:13,388] INFO [Worker clientId=connect-1, groupId=streaming-connect] Successfully joined group with generation 18535 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:469)
> [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1, groupId=streaming-connect] Joined group at generation 18535 and got assignment: Assignment{error=0, leader='connect-1-3580fdcc-257e-40e7-8243-f7839021599f', leaderUrl='http://10.13.191.32:8083/', offset=9164, connectorIds=[], taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1397)
> [2020-03-16 10:27:13,389] WARN [Worker clientId=connect-1, groupId=streaming-connect] Catching up to assignment's config offset. (org.apache.kafka.connect.runtime.distributed.DistributedHerder:909)
> [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1, groupId=streaming-connect] Current config state offset 12 is behind group assignment 9164, reading to end of config log (org.apache.kafka.connect.runtime.distributed.DistributedHerder:970)
>
>
>
> Thanks,
>
> Mandeep Gandhi
>
>
>
>
Re: Connect - Membership Protocol
Posted by Robin Moffatt <ro...@confluent.io>.
Are the four brokers across two clusters (B1+B2) / (B3+B4), or one cluster?
If one cluster, are using the same config/offset/status topics for each
Connect cluster? Because that definitely won't work.
In case it's useful:
https://rmoff.net/2019/11/22/common-mistakes-made-when-configuring-multiple-kafka-connect-workers/
--
Robin Moffatt | Senior Developer Advocate | robin@confluent.io | @rmoff
On Tue, 17 Mar 2020 at 17:58, mandeep gandhi <we...@gmail.com>
wrote:
> Gentle reminder.
>
> + users.
>
> On Mon, Mar 16, 2020 at 11:27 PM mandeep gandhi <we...@gmail.com>
> wrote:
>
> > Hi,
> >
> > (PS - I did not use users list as this concerns some internals)
> >
> > I was trying to deploy the following config on a K8s cluster -
> >
> > Namespace - X1 Connect group id - G1 Bootstrap servers - B1, B2
> > Namespace - X2 Connect group id - G1 Bootstrap servers - B3, B4
> >
> > With this configuration, I am seeing multiple rebalances (logs below). So
> > the question is how is the Membership protocol working?
> > I have tried to follow some KIPs and they mostly say that the Bootstrap
> > server is the coordinator of the group. If yes, then logically this
> > configuration should work just fine as both configs have different
> > bootstrap servers. But as far as I have tried to understand the code
> (and
> > tried to run Kafka Connect), it seems like the members get added in the
> > group one by one and the head of the list becomes the group
> co-ordinator. (
> > if I change Connect Group id in 2nd config, things work)
> >
> > Also, I wanted some pointers on what is happening at the server-side vs
> > client-side during the protocol.
> >
> >
> > Kindly help.
> >
> > Logs -
> >
> > [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Current config state offset 12 does not match
> group assignment 9164. Forcing rebalance.
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:942)
> > [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Rebalance started
> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:213)
> > [2020-03-16 10:27:13,386] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] (Re-)joining group
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:505)
> > [2020-03-16 10:27:13,388] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Successfully joined group with generation 18535
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:469)
> > [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Joined group at generation 18535 and got
> assignment: Assignment{error=0,
> leader='connect-1-3580fdcc-257e-40e7-8243-f7839021599f', leaderUrl='
> http://10.13.191.32:8083/', offset=9164, connectorIds=[], taskIds=[],
> revokedConnectorIds=[], revokedTaskIds=[], delay=0}
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1397)
> > [2020-03-16 10:27:13,389] WARN [Worker clientId=connect-1,
> groupId=streaming-connect] Catching up to assignment's config offset.
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:909)
> > [2020-03-16 10:27:13,389] INFO [Worker clientId=connect-1,
> groupId=streaming-connect] Current config state offset 12 is behind group
> assignment 9164, reading to end of config log
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:970)
> >
> >
> >
> > Thanks,
> >
> > Mandeep Gandhi
> >
> >
> >
> >
>