Leader became -1 and no ISR for all topics/patitions

[Karthi SsiSamsung <ka...@gmail.com>]

Hi,

I am facing similar issue where couple of my partitions have a Leader as -1
and No ISR. I tried to use kafka-reassign-partitions.sh and
kafka-preferred-replica-election.sh tool and this did not help as the ISR
was empty. Other users who faced this issue suggesting broker restart.

Does any one has other solutions of gracefully handling this instead of
restarting the brokers?

Re: Leader became -1 and no ISR for all topics/patitions

[Raju Bairishetti <ra...@apache.org>]

We had seen issue earlier in our production. We had some findings after
debugging

http://mail-archives.apache.org/mod_mbox/kafka-users/201603.mbox/%3CCAC2sYS16_yb-=FaL4mGtV7jKEWdLBVKXvdiZj_R8RuOAFEZqnA@mail.gmail.com%3E


On Thu, Nov 10, 2016 at 5:03 PM, Kevin A <k4...@gmail.com> wrote:

> Something similar happened to one of our clusters over the weekend. I'm
> still not sure how we got into that state. Running 0.9.0.1. Restarting the
> controller broker got us back on our feet.
>
> Slight difference is a few of our offline partitions had a single replica
> in the ISR list - but it wasn't the preferred replica. Running the
> reassignment tool to move the remaining ISR to the preferred position fixed
> most of those cases. A few, however, refused to budge. It was at this point
> I bounced the controller and everything went back to normal.
>
> On Wed, Nov 9, 2016 at 4:13 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Mind sharing how you got to this situation?
> >
> > Did you restart brokers? did the replicas drop off the ISR one by one
> > or all together? Do you have unclear leader election enabled? What are
> > the errors you are seeing in the logs?
> >
> > Once none of the partitions are available and lacking any additional
> > information, restarting the controller is probably the best choice,
> > followed by a complete restart of everything... but your cluster
> > really shouldn't get there.
> >
> >
> > On Wed, Nov 9, 2016 at 2:08 PM, Karthi SsiSamsung
> > <ka...@gmail.com> wrote:
> > > Hi,
> > >
> > > I am facing similar issue where couple of my partitions have a Leader
> as
> > -1
> > > and No ISR. I tried to use kafka-reassign-partitions.sh and
> > > kafka-preferred-replica-election.sh tool and this did not help as the
> > ISR
> > > was empty. Other users who faced this issue suggesting broker restart.
> > >
> > > Does any one has other solutions of gracefully handling this instead of
> > > restarting the brokers?
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter | blog
> >
>



-- 

------
Thanks,
Raju Bairishetti,
www.lazada.com

Re: Leader became -1 and no ISR for all topics/patitions

[Kevin A <k4...@gmail.com>]

Something similar happened to one of our clusters over the weekend. I'm
still not sure how we got into that state. Running 0.9.0.1. Restarting the
controller broker got us back on our feet.

Slight difference is a few of our offline partitions had a single replica
in the ISR list - but it wasn't the preferred replica. Running the
reassignment tool to move the remaining ISR to the preferred position fixed
most of those cases. A few, however, refused to budge. It was at this point
I bounced the controller and everything went back to normal.

On Wed, Nov 9, 2016 at 4:13 PM, Gwen Shapira <gw...@confluent.io> wrote:

> Mind sharing how you got to this situation?
>
> Did you restart brokers? did the replicas drop off the ISR one by one
> or all together? Do you have unclear leader election enabled? What are
> the errors you are seeing in the logs?
>
> Once none of the partitions are available and lacking any additional
> information, restarting the controller is probably the best choice,
> followed by a complete restart of everything... but your cluster
> really shouldn't get there.
>
>
> On Wed, Nov 9, 2016 at 2:08 PM, Karthi SsiSamsung
> <ka...@gmail.com> wrote:
> > Hi,
> >
> > I am facing similar issue where couple of my partitions have a Leader as
> -1
> > and No ISR. I tried to use kafka-reassign-partitions.sh and
> > kafka-preferred-replica-election.sh tool and this did not help as the
> ISR
> > was empty. Other users who faced this issue suggesting broker restart.
> >
> > Does any one has other solutions of gracefully handling this instead of
> > restarting the brokers?
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: Leader became -1 and no ISR for all topics/patitions

[Gwen Shapira <gw...@confluent.io>]

Mind sharing how you got to this situation?

Did you restart brokers? did the replicas drop off the ISR one by one
or all together? Do you have unclear leader election enabled? What are
the errors you are seeing in the logs?

Once none of the partitions are available and lacking any additional
information, restarting the controller is probably the best choice,
followed by a complete restart of everything... but your cluster
really shouldn't get there.


On Wed, Nov 9, 2016 at 2:08 PM, Karthi SsiSamsung
<ka...@gmail.com> wrote:
> Hi,
>
> I am facing similar issue where couple of my partitions have a Leader as -1
> and No ISR. I tried to use kafka-reassign-partitions.sh and
> kafka-preferred-replica-election.sh tool and this did not help as the ISR
> was empty. Other users who faced this issue suggesting broker restart.
>
> Does any one has other solutions of gracefully handling this instead of
> restarting the brokers?



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog