[jira] [Resolved] (KAFKA-4712) kafka-reassign-partitions causes ISR to shrink

["Colin McCabe (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Colin McCabe resolved KAFKA-4712.
---------------------------------
    Resolution: Cannot Reproduce

This seems to be related to overall load on the cluster as a result of the reassignment.  Let's close unless there is more to investigate here.

> kafka-reassign-partitions causes ISR to shrink
> ----------------------------------------------
>
>                 Key: KAFKA-4712
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4712
>             Project: Kafka
>          Issue Type: Bug
>          Components: admin
>    Affects Versions: 0.10.1.1
>            Reporter: Andrew Holford
>            Priority: Major
>
> Hi,
> We have been doing some work on our kafka clusters recently which required us to shrink the number of brokers and then increase the number again once the work on the hosts was complete. 
> During the reassign partitions work to move data away from and back to the servers we have worked on, the ISR has been seen to drop below our min.insync.replicas value causing an outage to our data flow until the ISR increases again.
> From the documentation our understanding was that during the reassignment the number of replicas should increase to include the new ones and then decrease back to the replication factor, is this not the case? 
> The process we have followed is below:
> 1. generate a json which contains all our topics..
> for example something like...
> {"version":1,"topics":[{"topic":"topic1"},{"topic":"topic2"},{"topic":"topic3"}]}
> 2. use the topic list json with the kafka-partition-assignment tool with the generate option, eg:
> bin/kafka-reassign-partitions --zookeeper zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 --topics-to-move-json-file /tmp/topic-list.json --broker-list "1,2,3,4,5,6,7" --generate
> 3. we then take the proposed configuration, place in another file, and execute as below:
> bin/kafka-reassign-partitions --zookeeper zookeeper-1:2181,zookeeper-2:2181,zookeeper-3:2181 --reassignment-json-file /tmp/topic-to-reassign.json --execute
> During the execution, we see lots of times different topics and different partitions reduce to 1 ISR, one example below:
> [2017-01-30 12:22:38,748] INFO Partition [com_ig_trade_v1_position_snapshot--live,9] on broker 5: Expanding ISR for partition [com_ig_trade_v1_position_snapshot--live,9] from 5,1 to 5,1,7 (kafka.cluster.Partition)
> [2017-01-30 12:22:38,712] INFO Partition [com_ig_trade_v1_position_snapshot--live,9] on broker 5: Expanding ISR for partition [com_ig_trade_v1_position_snapshot--live,9] from 5 to 5,1 (kafka.cluster.Partition)
> [2017-01-30 12:22:38,582] INFO Partition [com_ig_trade_v1_position_snapshot--live,9] on broker 5: Shrinking ISR for partition [com_ig_trade_v1_position_snapshot--live,9] from 5,1,7 to 5 (kafka.cluster.Partition)
> [2017-01-30 12:22:38,326] INFO Partition [com_ig_trade_v1_position_snapshot--live,9] on broker 2: Expanding ISR for partition [com_ig_trade_v1_position_snapshot--live,9] from 2,6,1,5 to 5,1,6,2,7 (kafka.cluster.Partition)
> [2017-01-30 12:14:49,906] INFO Partition [com_ig_trade_v1_position_snapshot--live,9] on broker 2: Expanding ISR for partition [com_ig_trade_v1_position_snapshot--live,9] from 2,6,1 to 2,6,1,5 (kafka.cluster.Partition)
> Is this what is expected?
> Do you have any recommendations / procedure / best practice which we could use when decreasing / increasing the cluster size, without the ISRs shrinking?
> Thanks,
> Andy. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)