Scaling Kafka Cluster in Cloud

[Varun Kumar <va...@hotmail.com>]

Hi,

I have a Kafka cluster setup in cloud, I tried a different approach to scale it.

I gracefully shutdown a broker with 2 (or more) disks and attach 2 brokers with 1 (or equally distributed) disks each. (Had to use partition reassignment tool change partitions after new broker is attached).

It actually worked with Kafka 0.11 where after reassigning partition, only the new data was copied to broker and broker added to ISR much faster than the approach given in documentation<https://kafka.apache.org/documentation/#basic_ops_cluster_expansion>. I understand for HA I need at least 3 replicas and all partitions in broker which will be scaled needs to have atleast 2 ISR other than the scaling broker.

Would like to know if this approach would work in production ? Any scenarios where data might be to data loss ? And also if it will actually work in latest version of Kafka ?

Thanks
Varun