You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by rk vishu <ta...@gmail.com> on 2013/10/01 22:17:09 UTC

Recommendation for number of brokers on kafka(0.7.2) hosts

Hello All,

I am currently using 5 node kafka cluster with 0.7.2 version. Would like to
get some advice on optimal number of brokers on each kafka host. Below is
the specification of each machine

- 4 data directories /data1,/data2, /data3, /data4 with 200+GB usable
space. RAID10
- 24 Core CPU
- 32GB RAM

>From the server properties, i believe we can only configure one data
log.dir.
So below are my questions
1) Will setting 4 brokers per host with different ports and different log
data directories be beneficial to use all the available space?
2) Will there be any disadvantage using multiple brokers on same host?
3) If partition count is set to 2 per topic. Having multiple brokers on
each machine , multiply the partition count per machine? i.e with 4 brokers
per machine on 5 machines will generate 2*4*5 = 40 partitions per topic?
4) Is there a way to use all the directories from a single broker?

Thanks and Regards
Ravi

Re: Recommendation for number of brokers on kafka(0.7.2) hosts

Posted by rk vishu <ta...@gmail.com>.
Thank you Neha for the suggestion.


On Tue, Oct 1, 2013 at 1:50 PM, Neha Narkhede <ne...@gmail.com>wrote:

> 1) Will setting 4 brokers per host with different ports and different log
> data directories be beneficial to use all the available space?
> 2) Will there be any disadvantage using multiple brokers on same host?
>
> It is recommended that you do not deploy multiple brokers on the same box
> since that will not lead to optimal page cache usage, which will affect IO
> performance.
>
> 3) If partition count is set to 2 per topic. Having multiple brokers on
> each machine , multiply the partition count per machine? i.e with 4 brokers
> per machine on 5 machines will generate 2*4*5 = 40 partitions per topic?
>
> Correct.
>
> 4) Is there a way to use all the directories from a single broker?
>
> Kafka 0.8 supports specifying multiple log directories through the log.dirs
> config.
>
> Thanks,
> Neha
>
>
> On Tue, Oct 1, 2013 at 1:17 PM, rk vishu <ta...@gmail.com> wrote:
>
> > Hello All,
> >
> > I am currently using 5 node kafka cluster with 0.7.2 version. Would like
> to
> > get some advice on optimal number of brokers on each kafka host. Below is
> > the specification of each machine
> >
> > - 4 data directories /data1,/data2, /data3, /data4 with 200+GB usable
> > space. RAID10
> > - 24 Core CPU
> > - 32GB RAM
> >
> > From the server properties, i believe we can only configure one data
> > log.dir.
> > So below are my questions
> > 1) Will setting 4 brokers per host with different ports and different log
> > data directories be beneficial to use all the available space?
> > 2) Will there be any disadvantage using multiple brokers on same host?
> > 3) If partition count is set to 2 per topic. Having multiple brokers on
> > each machine , multiply the partition count per machine? i.e with 4
> brokers
> > per machine on 5 machines will generate 2*4*5 = 40 partitions per topic?
> > 4) Is there a way to use all the directories from a single broker?
> >
> > Thanks and Regards
> > Ravi
> >
>

Re: Recommendation for number of brokers on kafka(0.7.2) hosts

Posted by Neha Narkhede <ne...@gmail.com>.
1) Will setting 4 brokers per host with different ports and different log
data directories be beneficial to use all the available space?
2) Will there be any disadvantage using multiple brokers on same host?

It is recommended that you do not deploy multiple brokers on the same box
since that will not lead to optimal page cache usage, which will affect IO
performance.

3) If partition count is set to 2 per topic. Having multiple brokers on
each machine , multiply the partition count per machine? i.e with 4 brokers
per machine on 5 machines will generate 2*4*5 = 40 partitions per topic?

Correct.

4) Is there a way to use all the directories from a single broker?

Kafka 0.8 supports specifying multiple log directories through the log.dirs
config.

Thanks,
Neha


On Tue, Oct 1, 2013 at 1:17 PM, rk vishu <ta...@gmail.com> wrote:

> Hello All,
>
> I am currently using 5 node kafka cluster with 0.7.2 version. Would like to
> get some advice on optimal number of brokers on each kafka host. Below is
> the specification of each machine
>
> - 4 data directories /data1,/data2, /data3, /data4 with 200+GB usable
> space. RAID10
> - 24 Core CPU
> - 32GB RAM
>
> From the server properties, i believe we can only configure one data
> log.dir.
> So below are my questions
> 1) Will setting 4 brokers per host with different ports and different log
> data directories be beneficial to use all the available space?
> 2) Will there be any disadvantage using multiple brokers on same host?
> 3) If partition count is set to 2 per topic. Having multiple brokers on
> each machine , multiply the partition count per machine? i.e with 4 brokers
> per machine on 5 machines will generate 2*4*5 = 40 partitions per topic?
> 4) Is there a way to use all the directories from a single broker?
>
> Thanks and Regards
> Ravi
>