Does Kafka support federation of clusters?

[高 志明 <gz...@hotmail.com>]

Hi,


For higher availability and more flexible operations (upgrading, rolling ops, etc.), we are considering to federate two or more  kafka clusters, each with a separated zk cluster. For example, if "topic-T" has 6 total partitions, 3 on cluster A, and another 3 on cluster B, messages will be partitioned (by key or random) into all 6 partitions ([A,0-2], [B,0-2]). On the other side, consumers have to accept two metadata broker lists for both clusters, process messages and store offsets separately.


Pros:

- more HA

- fewer loss for single broker machine down

- fewer loss when a network switch has problems or a rack is power off

- do green/blue compatible upgrading, and afford crashes of one entire cluster (maybe human mistakes, maybe bugs)

- with a special library of producers and consumers, we can do incompatible upgrading such as 0.7 -> 0.8


Cons:

- complex operations

- need wrap a special version of producers and consumers APIs for multiple clusters


Do you think this is a proper solution? Does Kafka have some similar features?



Thanks.