You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Scott Arthur <sa...@salesforce.com> on 2013/08/02 20:24:32 UTC

Relative cluster sizes and cluster size limits

Hi,

I have a question about scaling the broker count of a Kafka cluster.  We
have a scenario where we'll have two clusters replicating data into a
third.  We're wondering how we should size that third cluster so that it
can handle the volume of messages from the two source clusters.  Should we
just make the number of brokers match? e.g. five brokers in the two source
clusters, therefore 10 in the destination cluster.  In general, what is the
horizontal scaling model we should use?  Also, is there an upper limit to
the number of brokers you should put in a cluster, after which you get
diminishing returns on throughput?

Thanks,
Scott Arthur