What is viable hardware for Zookeeper and Kafka brokers?

[Mich Talebzadeh <mi...@gmail.com>]

Hi,

I have read in the note below

https://www.quora.com/What-is-viable-hardware-for-Zookeeper-and-Kafka-brokers

Which states

"Take your expected message size * expected messages/second, and multiply
that by how many seconds you would like to keep your messages available in
your kafka cluster for. Then multiply that size by two or more depending on
what you choose your replication factor (ie: redundancy) to be, and divide
that by how many brokers you'll have in your cluster. This will give you
how much hard drive space per broker that you need.

Kafka is a sequential message queue - there aren't any random reads and
writes. This means that it can take heavy advantage of disk read-ahead,
disk buffering, and linux kernel/filesystem optimizations. A typical hard
disk doing sequential reads/writes is easily able to saturate a gigabit
link, and so is Kafka with any decent CPU made in the last 5 years. This
means that you can get by with big, cheap, slow disks.

Please remember that the disk buffer cache lives in your RAM. That means
that you need sufficient RAM in order to store a certain time period of
messages in cache. If you're not doing anything crazy and are always
expecting your consumers to be up to date, then you don't have to worry too
much. Just keep this in mind should your consumers fall behind far enough
that they need to retrieve data outside of buffer cache. I personally would
use the money saved on disk to buy more RAM."


This pretty old (May 2015) and I am wondering if it refers to Kafka < 0.8).
However, the calculation's seems to be plausible. Any comments on it please?

Thanks


Dr Mich Talebzadeh



LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.