Kafka - replicas do not heal themselves by default

[Nag Y <an...@gmail.com>]

I am going through the kafka in action and come across this following phrase

*One of the things to note with Kafka is that replicas do not heal
themselves by default. If you lose a broker on which one of your copies of
a partition existed, Kafka does not currently create a new copy. I mention
this since some users are used to filesystems like HDFS that will maintain
that replication number if a block is seen as corrupted or failed. So an
important item to look at with monitoring the health of your system might
be how many of your ISRs are indeed matching your intended number.*


It looks interesting, as in most of the distributed systems, systems will
try to create additional replicas if replicas are not available. I found it
strange,  Any reason to do so ?

Re: Kafka - replicas do not heal themselves by default

[Nag Y <an...@gmail.com>]

I assume, in the case of kubernetes, Auto healing is something provided by
the kubernetes and not by Kafka.

Isn't it?

My intention of asking the question is, what is the reasoning behind it?

Version is Kafka 2.5, you can observe this with latest confluent version as
well

On Mon, 15 Jun, 2020, 3:01 AM Israel Ekpo, <is...@gmail.com> wrote:

> It is always good to have context. It would be helpful to state the edition
> of the book, version of Kafka, deployment architecture and other
> environment details
>
> What edition of the book are you referring to?
>
> What version of Kafka is used in the book.
>
> How the producers and consumers are interacting with the brokers influences
> what happens and the impact.
>
> The project is rapidly evolving and if you are running on Kubernetes, the
> self healing aspect happens automatically in my experience for most use
> cases
>
> Could you share more details for the actual scenarios you are working on
> outside the book?
>
>
>
> On Sun, Jun 14, 2020 at 1:56 PM Nag Y <an...@gmail.com> wrote:
>
> > I am going through the kafka in action and come across this following
> > phrase
> >
> > *One of the things to note with Kafka is that replicas do not heal
> > themselves by default. If you lose a broker on which one of your copies
> of
> > a partition existed, Kafka does not currently create a new copy. I
> mention
> > this since some users are used to filesystems like HDFS that will
> maintain
> > that replication number if a block is seen as corrupted or failed. So an
> > important item to look at with monitoring the health of your system might
> > be how many of your ISRs are indeed matching your intended number.*
> >
> >
> > It looks interesting, as in most of the distributed systems, systems will
> > try to create additional replicas if replicas are not available. I found
> it
> > strange,  Any reason to do so ?
> >
>

Re: Kafka - replicas do not heal themselves by default

[Israel Ekpo <is...@gmail.com>]

It is always good to have context. It would be helpful to state the edition
of the book, version of Kafka, deployment architecture and other
environment details

What edition of the book are you referring to?

What version of Kafka is used in the book.

How the producers and consumers are interacting with the brokers influences
what happens and the impact.

The project is rapidly evolving and if you are running on Kubernetes, the
self healing aspect happens automatically in my experience for most use
cases

Could you share more details for the actual scenarios you are working on
outside the book?



On Sun, Jun 14, 2020 at 1:56 PM Nag Y <an...@gmail.com> wrote:

> I am going through the kafka in action and come across this following
> phrase
>
> *One of the things to note with Kafka is that replicas do not heal
> themselves by default. If you lose a broker on which one of your copies of
> a partition existed, Kafka does not currently create a new copy. I mention
> this since some users are used to filesystems like HDFS that will maintain
> that replication number if a block is seen as corrupted or failed. So an
> important item to look at with monitoring the health of your system might
> be how many of your ISRs are indeed matching your intended number.*
>
>
> It looks interesting, as in most of the distributed systems, systems will
> try to create additional replicas if replicas are not available. I found it
> strange,  Any reason to do so ?
>

Re: Kafka - replicas do not heal themselves by default

[Israel Ekpo <is...@gmail.com>]

It is always good to have context. It would be helpful to state the edition
of the book, version of Kafka, deployment architecture and other
environment details

What edition of the book are you referring to?

What version of Kafka is used in the book.

How the producers and consumers are interacting with the brokers influences
what happens and the impact.

The project is rapidly evolving and if you are running on Kubernetes, the
self healing aspect happens automatically in my experience for most use
cases

Could you share more details for the actual scenarios you are working on
outside the book?



On Sun, Jun 14, 2020 at 1:56 PM Nag Y <an...@gmail.com> wrote:

> I am going through the kafka in action and come across this following
> phrase
>
> *One of the things to note with Kafka is that replicas do not heal
> themselves by default. If you lose a broker on which one of your copies of
> a partition existed, Kafka does not currently create a new copy. I mention
> this since some users are used to filesystems like HDFS that will maintain
> that replication number if a block is seen as corrupted or failed. So an
> important item to look at with monitoring the health of your system might
> be how many of your ISRs are indeed matching your intended number.*
>
>
> It looks interesting, as in most of the distributed systems, systems will
> try to create additional replicas if replicas are not available. I found it
> strange,  Any reason to do so ?
>