Reassign partitions failures

[Anderson Goulart <an...@boxever.com>]

Hi,

We are having a issue with our kafka cluster when we try to reassign 
partitions. Imagine the following topic layout file:

Kafka Version: 0.8.2

{
     "partitions": [
         {
             "partition": 0,
             "replicas": [
                 1,
                 2,
                 3
             ],
             "topic": "mytopic"
         },
     ],
     "version": 1
}

And imagine we want to change replicas broker ids 2 and 3 to 4 and 5. 
After changing it, the final layout would be:

{
     "partitions": [
         {
             "partition": 0,
             "replicas": [
                 5,
                 1,
                 4
             ],
             "topic": "mytopic"
         },
     ],
     "version": 1
}


After running

./bin/kafka-reassign-partitions.sh --zookeeper $zookeeper 
--reassignment-json-file /tmp/mytopic --execute we got the error

*ERROR: Assigned replicas (*1*,*2*,*3*,*4*,*5*) don't match the list of 
replicas for reassignment (*5,1,4)* for partition [*mytopic*,0]*

It failed and its behavior is really strange:

  * It increased the replication factor to 5 instead of keeping 3 having
    a replica list [1, 2, 3, 4, 5]  and ISR (514)
  * Number 4 became the new leader (which is ok)
  * The in-sync replica only has 3 ISRs instead of 5
  * When we tried to restart the node 1, we lost it. It started
    smoothly, but it could rejoin the cluster and it stopped appearing
    on ISR list

Questions:

1) Why it increased the replica list?

2) Any ideas of how to investigate the problem?