Partition reassignment: partition does not exist

[Alexis Lesieur <al...@signal.co>]

Hi!

I'm trying to increase the replication factor on all my topics (from 2 to 3), and I am currently facing a wall...

So here is what I had:
- 2 brokers (indexed starting at 0)
- ~300 topics, most of them with 1 partition (no need to argue that point, that's how it is, and that's deliberate)
- all of them with a replication factor of 2 (so assigned to brokers 0 and 1, and ISR)

What I did:
- created a 3rd broker
- updated the brokers configs to have:
default.replication.factor=3
offsets.topic.replication.factor=3
- restarted said brokers
- generated a reassignment file json, which basically looked like this:
```{"partitions":[
{"topic": "topic1 ", "partition": 0, "replicas": [0,1,2]},
{"topic": "topic2 ", "partition": 0, "replicas": [0,1,2]},
{"topic": "topic3 ", "partition": 0, "replicas": [0,1,2]},
...
{"topic": "topicXYZ ", "partition": 0, "replicas": [0,1,2]}
],"version":1}```
- ran the following:
`bin/kafka-reassign-partitions.sh --zookeeper zk_host:2181/chroot --reassignment-json-file ~/assignment.json --execute`

And... I got some of these (~30):
[2016-05-10 20:23:18,205] ERROR Skipping reassignment of partition [topicXXX ,0] since it doesn't exist (kafka.admin.ReassignPartitionsCommand)
ERROR: Assigned replicas () don't match the list of replicas for reassignment (0,1,2) for partition [topicXXX ,0]

The partitions do exist, I can do a --describe --topic on them, we are consuming from them, ... Just the reassignment tool somehow apparently believes they don't exist.
We could spot nothing in the logs either.

Any tips on what I can do to fix this?

Thank you very much!
Alexis

PS: I tried, for one of these topic, to do a preferred leader election. As far as I can tell, the leader did not change, and an attempt to assign all 3 brokers as replicas for that topic fails.
PPS: The assignment has been attempted from both the new broker, and an "old" one.
PPPS: Please forgive any mistakes, english is not my native language :-)