You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Paolo Patierno <pp...@live.com> on 2016/05/09 16:03:22 UTC

New broker as follower for a topic replication

Hello,

in order to try running Kafka in Kubernetes I'm facing the following problem ...

Imagine that I start a cluster with a zookeeper instance and two kafka broker. I create a topic with one partition but replication factor 2 : I have broker1 as leader and broker2 as follower.
All works great !

Imagine that the pod where the broker2 is running stops to work. Kubernetes starts a new pod (let me name it broker3) but with a different IP. 
What's the right way to make the new broker instance as a follower for the broker1 in order to have it "in sync" and guarantee the replication factor 2 for the topic ?

Thanks,
Paolo.

Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor 
Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience 		 	   		  

Re: New broker as follower for a topic replication

Posted by Alex Loddengaard <al...@confluent.io>.
Hi Paolo,

The best way to do this would be to have broker3 start up with the same
broker id as the failed broker2. broker3 will then rejoin the cluster,
begin catching up with broker1, and eventually rejoin the ISR. If it starts
up with a new broker id, you'll need to run the partition reassignment tool
to manually assign partition replicas to the new broker.

Alex

On Mon, May 9, 2016 at 9:03 AM, Paolo Patierno <pp...@live.com> wrote:

> Hello,
>
> in order to try running Kafka in Kubernetes I'm facing the following
> problem ...
>
> Imagine that I start a cluster with a zookeeper instance and two kafka
> broker. I create a topic with one partition but replication factor 2 : I
> have broker1 as leader and broker2 as follower.
> All works great !
>
> Imagine that the pod where the broker2 is running stops to work.
> Kubernetes starts a new pod (let me name it broker3) but with a different
> IP.
> What's the right way to make the new broker instance as a follower for the
> broker1 in order to have it "in sync" and guarantee the replication factor
> 2 for the topic ?
>
> Thanks,
> Paolo.
>
> Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat
> Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor
> Twitter : @ppatierno
> Linkedin : paolopatierno
> Blog : DevExperience