You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by SenthilKumar K <se...@gmail.com> on 2019/11/12 07:33:48 UTC
Moving partition(s) to different broker
Hi Experts, We have seen a problem with partition leader i.e it's set to -1.
describe o/p:
Topic: 1453 Partition: 47 Leader: -1 Replicas: 24,15 Isr: 24
Kafka Version: 2.2.0
Replication: 2
Partitions: 48
Brokers 24 ,15 both are down due to disk errors and we lost the partition
47. I tried increasing the replica ( 2 to 3 ) of the partition alone using
the Kafka partition reassignment tool but that didn't help.
{
"version": 1,
"partitions": [{
"topic": "1453",
"partition": 47,
"replicas": [22, 11, 5],
"log_dirs": ["any", "any", "any"]
}]
}
Reassignment of partition 1453-47 is still in progress - Its stuck more
than 3 hours.
How to recover the partition 47? Pls advise. Thanks!
--Senthil
Re: Moving partition(s) to different broker
Posted by SenthilKumar K <se...@gmail.com>.
Thanks, Peter for the quick response.
You may need to enable unclean leader election, as well. - We already set
unclean leader election to true. Unfortunately both brokers went down
within 10 minutes of time. Another option is to increase the replication
factor to 3 from 2. Right now the goal is to have no offline partitions in
a cluster.
Is your goal just to have no offline partitions or to recover the data
contained in the affected partition? - Our goal is to have no offline
partitions. We are fine with losing the data of affected partition.
--Senthil
On Tue, Nov 12, 2019 at 1:18 PM Peter Bukowinski <pm...@gmail.com> wrote:
> If the only replicas for that topic partition exist on brokers 15 and 24
> and they are both down, then you cannot recover the partition until either
> of them is replaced or repaired and rejoins the cluster. You may need to
> enable unclean leader election, as well.
>
> As you’ve discovered, adding replicas to an offline partition partition
> doesn’t work. A topic/partition needs to be in a healthy state for that to
> work.
>
> Is your goal just to have no offline partitions or to recover the data
> contained in the affected partition? Producers and consumers should still
> be able to access the topic in its current state.
>
> --
> Peter
>
> > On Nov 11, 2019, at 11:34 PM, SenthilKumar K <se...@gmail.com>
> wrote:
> >
> > Hi Experts, We have seen a problem with partition leader i.e it's set
> to -1.
> >
> > describe o/p:
> > Topic: 1453 Partition: 47 Leader: -1 Replicas: 24,15 Isr: 24
> >
> > Kafka Version: 2.2.0
> > Replication: 2
> > Partitions: 48
> >
> > Brokers 24 ,15 both are down due to disk errors and we lost the partition
> > 47. I tried increasing the replica ( 2 to 3 ) of the partition alone
> using
> > the Kafka partition reassignment tool but that didn't help.
> >
> > {
> >
> > "version": 1,
> >
> > "partitions": [{
> >
> > "topic": "1453",
> >
> > "partition": 47,
> >
> > "replicas": [22, 11, 5],
> >
> > "log_dirs": ["any", "any", "any"]
> >
> > }]
> >
> > }
> >
> >
> > Reassignment of partition 1453-47 is still in progress - Its stuck more
> > than 3 hours.
> >
> >
> > How to recover the partition 47? Pls advise. Thanks!
> >
> > --Senthil
>
Re: Moving partition(s) to different broker
Posted by Peter Bukowinski <pm...@gmail.com>.
If the only replicas for that topic partition exist on brokers 15 and 24 and they are both down, then you cannot recover the partition until either of them is replaced or repaired and rejoins the cluster. You may need to enable unclean leader election, as well.
As you’ve discovered, adding replicas to an offline partition partition doesn’t work. A topic/partition needs to be in a healthy state for that to work.
Is your goal just to have no offline partitions or to recover the data contained in the affected partition? Producers and consumers should still be able to access the topic in its current state.
--
Peter
> On Nov 11, 2019, at 11:34 PM, SenthilKumar K <se...@gmail.com> wrote:
>
> Hi Experts, We have seen a problem with partition leader i.e it's set to -1.
>
> describe o/p:
> Topic: 1453 Partition: 47 Leader: -1 Replicas: 24,15 Isr: 24
>
> Kafka Version: 2.2.0
> Replication: 2
> Partitions: 48
>
> Brokers 24 ,15 both are down due to disk errors and we lost the partition
> 47. I tried increasing the replica ( 2 to 3 ) of the partition alone using
> the Kafka partition reassignment tool but that didn't help.
>
> {
>
> "version": 1,
>
> "partitions": [{
>
> "topic": "1453",
>
> "partition": 47,
>
> "replicas": [22, 11, 5],
>
> "log_dirs": ["any", "any", "any"]
>
> }]
>
> }
>
>
> Reassignment of partition 1453-47 is still in progress - Its stuck more
> than 3 hours.
>
>
> How to recover the partition 47? Pls advise. Thanks!
>
> --Senthil