You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Alok Dwivedi <al...@maersk.com> on 2019/09/13 10:51:02 UTC
guidelines for replacing a lost Kafka Broker
Hi All
Can someone please advise what is the recommended practice for replacing a lost broker in a Kafka cluster. Lets consider this sequence of events
1. 3 node cluster say with 1 topic of 3 partitions and RF of 2.
2. That gives 2 partitions on each Broker (B1,B2,B3)
3. Lets say we lost B3. We still have B1 and B2 and 4 partitions. Assuming each broker had only 1 leader, by losing B3 we lost 1 leader and 1 replica. Out of remaining replicas another leader will be selected for any lost leaders so we carry on with no downtime.
4. I want to know what we should do to go back to 3 broker cluster. Are there standard guidelines around any procedure to be done after restarting the lost broker mainly when the lost broker has been replaced with another instance which no longer has old log.dirs i.e. data existing on this lost broker is gone. Is there a way to rebuild missing replica partitions now from existing partitions or what is the recommended practice in that scenario? Will we need some kind of partition re-assignment here and will it ensure that we now go back from 4 partitions to 6 partitions?
Can someone please advise what happens automatically in this case and what (if anything) should be done as a standard practice to get things back to normalcy.
Thanks
Alok Dwivedi
Data Architect
Maersk Always On Platform
Classification: Public
________________________________
The information contained in this message is privileged and intended only for the recipients named. If the reader is not a representative of the intended recipient, any review, dissemination or copying of this message or the information it contains is prohibited. If you have received this message in error, please immediately notify the sender, and delete the original message and attachments.
Maersk will as part of our communication and interaction with you collect and process your personal data. You can read more about Maersk's collection and processing of your personal data and your rights as a data subject in our privacy policy <https://www.maersk.com/front-page-requirements/privacy-policy>
Please consider the environment before printing this email.
Re: guidelines for replacing a lost Kafka Broker
Posted by Svante Karlsson <sv...@csi.se>.
Just bring a new broker up and give it the id of the lost one. It will sync
itself
/svante
Den fre 13 sep. 2019 kl 13:51 skrev saurav suman <sa...@gmail.com>:
> Hi,
>
> When the old data is lost and another broker is added to the cluster then
> it is a new fresh broker with no data. You can reassign the partitions of
> the topics using kafka-reassign-partitions.sh script.
>
> Please check the below links for more details.
> https://blog.imaginea.com/how-to-rebalance-topics-in-kafka-cluster/
>
> https://kafka.apache.org/documentation/
>
> Regards,
> Saurav Suman
>
> On Fri, Sep 13, 2019 at 4:21 PM Alok Dwivedi <al...@maersk.com>
> wrote:
>
> > Hi All
> > Can someone please advise what is the recommended practice for replacing
> a
> > lost broker in a Kafka cluster. Lets consider this sequence of events
> >
> >
> > 1. 3 node cluster say with 1 topic of 3 partitions and RF of 2.
> > 2. That gives 2 partitions on each Broker (B1,B2,B3)
> > 3. Lets say we lost B3. We still have B1 and B2 and 4 partitions.
> > Assuming each broker had only 1 leader, by losing B3 we lost 1 leader
> and 1
> > replica. Out of remaining replicas another leader will be selected for
> any
> > lost leaders so we carry on with no downtime.
> > 4. I want to know what we should do to go back to 3 broker cluster.
> Are
> > there standard guidelines around any procedure to be done after
> restarting
> > the lost broker mainly when the lost broker has been replaced with
> another
> > instance which no longer has old log.dirs i.e. data existing on this lost
> > broker is gone. Is there a way to rebuild missing replica partitions now
> > from existing partitions or what is the recommended practice in that
> > scenario? Will we need some kind of partition re-assignment here and will
> > it ensure that we now go back from 4 partitions to 6 partitions?
> >
> >
> > Can someone please advise what happens automatically in this case and
> what
> > (if anything) should be done as a standard practice to get things back to
> > normalcy.
> >
> > Thanks
> > Alok Dwivedi
> > Data Architect
> > Maersk Always On Platform
> >
> >
> >
> >
> >
> > Classification: Public
> >
> > ________________________________
> >
> > The information contained in this message is privileged and intended only
> > for the recipients named. If the reader is not a representative of the
> > intended recipient, any review, dissemination or copying of this message
> or
> > the information it contains is prohibited. If you have received this
> > message in error, please immediately notify the sender, and delete the
> > original message and attachments.
> >
> > Maersk will as part of our communication and interaction with you collect
> > and process your personal data. You can read more about Maersk's
> collection
> > and processing of your personal data and your rights as a data subject in
> > our privacy policy <
> > https://www.maersk.com/front-page-requirements/privacy-policy>
> >
> > Please consider the environment before printing this email.
> >
>
>
> --
>
> Thanks & Regards,
>
> Saurav Suman
> MOB - 8884222451
>
Re: guidelines for replacing a lost Kafka Broker
Posted by saurav suman <sa...@gmail.com>.
Hi,
When the old data is lost and another broker is added to the cluster then
it is a new fresh broker with no data. You can reassign the partitions of
the topics using kafka-reassign-partitions.sh script.
Please check the below links for more details.
https://blog.imaginea.com/how-to-rebalance-topics-in-kafka-cluster/
https://kafka.apache.org/documentation/
Regards,
Saurav Suman
On Fri, Sep 13, 2019 at 4:21 PM Alok Dwivedi <al...@maersk.com>
wrote:
> Hi All
> Can someone please advise what is the recommended practice for replacing a
> lost broker in a Kafka cluster. Lets consider this sequence of events
>
>
> 1. 3 node cluster say with 1 topic of 3 partitions and RF of 2.
> 2. That gives 2 partitions on each Broker (B1,B2,B3)
> 3. Lets say we lost B3. We still have B1 and B2 and 4 partitions.
> Assuming each broker had only 1 leader, by losing B3 we lost 1 leader and 1
> replica. Out of remaining replicas another leader will be selected for any
> lost leaders so we carry on with no downtime.
> 4. I want to know what we should do to go back to 3 broker cluster. Are
> there standard guidelines around any procedure to be done after restarting
> the lost broker mainly when the lost broker has been replaced with another
> instance which no longer has old log.dirs i.e. data existing on this lost
> broker is gone. Is there a way to rebuild missing replica partitions now
> from existing partitions or what is the recommended practice in that
> scenario? Will we need some kind of partition re-assignment here and will
> it ensure that we now go back from 4 partitions to 6 partitions?
>
>
> Can someone please advise what happens automatically in this case and what
> (if anything) should be done as a standard practice to get things back to
> normalcy.
>
> Thanks
> Alok Dwivedi
> Data Architect
> Maersk Always On Platform
>
>
>
>
>
> Classification: Public
>
> ________________________________
>
> The information contained in this message is privileged and intended only
> for the recipients named. If the reader is not a representative of the
> intended recipient, any review, dissemination or copying of this message or
> the information it contains is prohibited. If you have received this
> message in error, please immediately notify the sender, and delete the
> original message and attachments.
>
> Maersk will as part of our communication and interaction with you collect
> and process your personal data. You can read more about Maersk's collection
> and processing of your personal data and your rights as a data subject in
> our privacy policy <
> https://www.maersk.com/front-page-requirements/privacy-policy>
>
> Please consider the environment before printing this email.
>
--
Thanks & Regards,
Saurav Suman
MOB - 8884222451
Re: guidelines for replacing a lost Kafka Broker
Posted by "M. Manna" <ma...@gmail.com>.
Not sure what you mean by 1 topic with 3 partition and replication-factor =
2. You may need to revisit the idea of partitions and replication factor
from docs.
Thanks,
On Fri, 13 Sep 2019 at 11:51, Alok Dwivedi <al...@maersk.com> wrote:
> Hi All
> Can someone please advise what is the recommended practice for replacing a
> lost broker in a Kafka cluster. Lets consider this sequence of events
>
>
> 1. 3 node cluster say with 1 topic of 3 partitions and RF of 2.
> 2. That gives 2 partitions on each Broker (B1,B2,B3)
> 3. Lets say we lost B3. We still have B1 and B2 and 4 partitions.
> Assuming each broker had only 1 leader, by losing B3 we lost 1 leader and 1
> replica. Out of remaining replicas another leader will be selected for any
> lost leaders so we carry on with no downtime.
> 4. I want to know what we should do to go back to 3 broker cluster. Are
> there standard guidelines around any procedure to be done after restarting
> the lost broker mainly when the lost broker has been replaced with another
> instance which no longer has old log.dirs i.e. data existing on this lost
> broker is gone. Is there a way to rebuild missing replica partitions now
> from existing partitions or what is the recommended practice in that
> scenario? Will we need some kind of partition re-assignment here and will
> it ensure that we now go back from 4 partitions to 6 partitions?
>
>
> Can someone please advise what happens automatically in this case and what
> (if anything) should be done as a standard practice to get things back to
> normalcy.
>
> Thanks
> Alok Dwivedi
> Data Architect
> Maersk Always On Platform
>
>
>
>
>
> Classification: Public
>
> ________________________________
>
> The information contained in this message is privileged and intended only
> for the recipients named. If the reader is not a representative of the
> intended recipient, any review, dissemination or copying of this message or
> the information it contains is prohibited. If you have received this
> message in error, please immediately notify the sender, and delete the
> original message and attachments.
>
> Maersk will as part of our communication and interaction with you collect
> and process your personal data. You can read more about Maersk's collection
> and processing of your personal data and your rights as a data subject in
> our privacy policy <
> https://www.maersk.com/front-page-requirements/privacy-policy>
>
> Please consider the environment before printing this email.
>