rule to set number of brokers for each server

[Sa Li <sa...@gmail.com>]

Dear all

I am provision production kafka cluster, which has 3 servers, I am
wondering how many brokers I should set for each servers, I set 3 brokers
in dev clusters, but I really don't what is the advantages to set more than
1 broker for each server, what about 1 broker for each server, totally 3
brokers, instead of 9 brokers.

thanks


-- 

Alec Li

Re: rule to set number of brokers for each server

[Joe Stein <jo...@stealth.ly>]

Unless you are using some containerized implementation setting more than
one broker on a server is likely going defeat some of the benefit Kafka
brings with how it leverages page cache (more on that here
https://kafka.apache.org/documentation.html#maximizingefficiency). You also
need to think about competing for the disk usage too.

If this is just for testing development then having more than 3 brokers
allows you to play with more failure and operation maintenance
https://kafka.apache.org/documentation.html#basic_ops scenarios (especially
if you set replication=3).

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/

On Fri, Nov 28, 2014 at 12:24 PM, Sa Li <sa...@gmail.com> wrote:

> Dear all
>
> I am provision production kafka cluster, which has 3 servers, I am
> wondering how many brokers I should set for each servers, I set 3 brokers
> in dev clusters, but I really don't what is the advantages to set more than
> 1 broker for each server, what about 1 broker for each server, totally 3
> brokers, instead of 9 brokers.
>
> thanks
>
>
> --
>
> Alec Li
>

Re: rule to set number of brokers for each server

[Gwen Shapira <gs...@cloudera.com>]

I don't see any advantage to more than one broker per server. In my
experience a single broker is capable of saturating the network link
and therefore I can't see how a second or third brokers will give any
benefits.

Gwen

On Fri, Nov 28, 2014 at 9:24 AM, Sa Li <sa...@gmail.com> wrote:
> Dear all
>
> I am provision production kafka cluster, which has 3 servers, I am
> wondering how many brokers I should set for each servers, I set 3 brokers
> in dev clusters, but I really don't what is the advantages to set more than
> 1 broker for each server, what about 1 broker for each server, totally 3
> brokers, instead of 9 brokers.
>
> thanks
>
>
> --
>
> Alec Li