Kafka write MB/s

[Dominik Werder <do...@gmail.com>]

Hello everyone,

I'm interested in the maximum transfer rate of a Kafka broker. There
is a fair bit of performance figures in terms of messages per second,
but I'm more interested in MB/s with rather large messages, let's say
500 kB as an example.

I tried testing by writing with the apache kafka performance test
tool, and with librdkafka.

The hardware can write continously 700 MB/s to local storage, but the
maximum I can achieve with Kafka is about 260 MB/s with acks=0.

I varied the various server side and client side options, batch sizes,
message sizes, send and receive buffer sizes, network buffer sizes,
JVM heap size. I tested both over loopback and over 10Gb ethernet,
same results.

Before digging into this much more, I wondered whether somebody has
experience with writing to a single Kafka broker at these transfer
rates?

Best regards
Dominik

Re: Kafka write MB/s

[SenthilKumar K <se...@gmail.com>]

I tried benchmarking kafka producer with acks=1 in 5 node cluster ..

Total transfer rate is ~950MB/sec .. Single broker transfer rate is less
than 200MB/sec..

Load Generator:
I've started 6 instance of http server where it writes to broker ... Using
wrk2 http benchmarking tool I was able to send more events to Kafka Cluster
for this benchmarking test..

--Senthil

On Jul 3, 2017 2:17 PM, "Dominik Werder" <do...@gmail.com> wrote:

Hello everyone,

I'm interested in the maximum transfer rate of a Kafka broker. There
is a fair bit of performance figures in terms of messages per second,
but I'm more interested in MB/s with rather large messages, let's say
500 kB as an example.

I tried testing by writing with the apache kafka performance test
tool, and with librdkafka.

The hardware can write continously 700 MB/s to local storage, but the
maximum I can achieve with Kafka is about 260 MB/s with acks=0.

I varied the various server side and client side options, batch sizes,
message sizes, send and receive buffer sizes, network buffer sizes,
JVM heap size. I tested both over loopback and over 10Gb ethernet,
same results.

Before digging into this much more, I wondered whether somebody has
experience with writing to a single Kafka broker at these transfer
rates?

Best regards
Dominik