You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ryan Berdeen <rb...@hubspot.com> on 2014/04/21 20:41:29 UTC
Too many replicas after partition reassignment
After doing some partition reassignments, I've ended up with some
partitions that have both the old and new brokers assigned.
The output of kafka-topics.sh --describe looks like this:
Topic:cs-es-indexer-a PartitionCount:30 ReplicationFactor:2 Configs:
retention.ms=10800000
...
Topic: cs-es-indexer-a Partition: 11 Leader: 11 Replicas: 11,12 Isr:
11,12
Topic: cs-es-indexer-a Partition: 12 Leader: 15 Replicas: 15,17,12
Isr: 12,17,15
...
Partition 12 has 3 replicas, but the topic has a replication factor of 2.
It was reassigned from [12,17] to [15,17].
During the reassignment, the controller changed from one broker to another,
and the resigning controller deadlocked with
https://issues.apache.org/jira/browse/KAFKA-1317.
Is this expected, or a known issue? I've had to alter some of my monitoring
tools to handle different-sized replica lists for partitions of the same
topic.
Re: Too many replicas after partition reassignment
Posted by Timothy Chen <tn...@gmail.com>.
Hi Ryan,
Also KAFKA-1317 should be fixed in both trunk and latest 0.8.1 branch,
are you running with either or just with one of the previous released
versions?
Tim
On Mon, Apr 21, 2014 at 5:00 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Ryan,
>
> Did you see any error logs on the new controller's controller log and
> state-change log?
>
> Guozhang
>
>
> On Mon, Apr 21, 2014 at 11:41 AM, Ryan Berdeen <rb...@hubspot.com> wrote:
>
>> After doing some partition reassignments, I've ended up with some
>> partitions that have both the old and new brokers assigned.
>>
>> The output of kafka-topics.sh --describe looks like this:
>>
>> Topic:cs-es-indexer-a PartitionCount:30 ReplicationFactor:2 Configs:
>> retention.ms=10800000
>> ...
>> Topic: cs-es-indexer-a Partition: 11 Leader: 11 Replicas: 11,12 Isr:
>> 11,12
>> Topic: cs-es-indexer-a Partition: 12 Leader: 15 Replicas: 15,17,12
>> Isr: 12,17,15
>> ...
>>
>> Partition 12 has 3 replicas, but the topic has a replication factor of 2.
>> It was reassigned from [12,17] to [15,17].
>>
>> During the reassignment, the controller changed from one broker to another,
>> and the resigning controller deadlocked with
>> https://issues.apache.org/jira/browse/KAFKA-1317.
>>
>> Is this expected, or a known issue? I've had to alter some of my monitoring
>> tools to handle different-sized replica lists for partitions of the same
>> topic.
>>
>
>
>
> --
> -- Guozhang
Re: Too many replicas after partition reassignment
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Ryan,
Did you see any error logs on the new controller's controller log and
state-change log?
Guozhang
On Mon, Apr 21, 2014 at 11:41 AM, Ryan Berdeen <rb...@hubspot.com> wrote:
> After doing some partition reassignments, I've ended up with some
> partitions that have both the old and new brokers assigned.
>
> The output of kafka-topics.sh --describe looks like this:
>
> Topic:cs-es-indexer-a PartitionCount:30 ReplicationFactor:2 Configs:
> retention.ms=10800000
> ...
> Topic: cs-es-indexer-a Partition: 11 Leader: 11 Replicas: 11,12 Isr:
> 11,12
> Topic: cs-es-indexer-a Partition: 12 Leader: 15 Replicas: 15,17,12
> Isr: 12,17,15
> ...
>
> Partition 12 has 3 replicas, but the topic has a replication factor of 2.
> It was reassigned from [12,17] to [15,17].
>
> During the reassignment, the controller changed from one broker to another,
> and the resigning controller deadlocked with
> https://issues.apache.org/jira/browse/KAFKA-1317.
>
> Is this expected, or a known issue? I've had to alter some of my monitoring
> tools to handle different-sized replica lists for partitions of the same
> topic.
>
--
-- Guozhang
Re: Too many replicas after partition reassignment
Posted by Jun Rao <ju...@gmail.com>.
Which version of Kafka are you using?
Thanks,
Jun
On Mon, Apr 21, 2014 at 11:41 AM, Ryan Berdeen <rb...@hubspot.com> wrote:
> After doing some partition reassignments, I've ended up with some
> partitions that have both the old and new brokers assigned.
>
> The output of kafka-topics.sh --describe looks like this:
>
> Topic:cs-es-indexer-a PartitionCount:30 ReplicationFactor:2 Configs:
> retention.ms=10800000
> ...
> Topic: cs-es-indexer-a Partition: 11 Leader: 11 Replicas: 11,12 Isr:
> 11,12
> Topic: cs-es-indexer-a Partition: 12 Leader: 15 Replicas: 15,17,12
> Isr: 12,17,15
> ...
>
> Partition 12 has 3 replicas, but the topic has a replication factor of 2.
> It was reassigned from [12,17] to [15,17].
>
> During the reassignment, the controller changed from one broker to another,
> and the resigning controller deadlocked with
> https://issues.apache.org/jira/browse/KAFKA-1317.
>
> Is this expected, or a known issue? I've had to alter some of my monitoring
> tools to handle different-sized replica lists for partitions of the same
> topic.
>