Tune Kafka cluster performance

[David Yu <gu...@gmail.com>]

Hi, Kafka,

We have a 2 broker staging cluster and a 6 broker prod cluster (all running
0.8.2.1). I'm trying to make sense of the producer performances on both.

Here are some perf test results:

-----------------

Prod:

kafka-producer-perf-test.sh --broker-list
kafkaProd-1:9092,kafkaProd-2:9092,kafkaProd-3:9092,kafkaProd-4:9092,kafkaProd-5:9092,kafkaProd-6:9092
--topics perf_test --message-size 2400 --messages 50000 --batch-size 200
--request-num-acks 1

start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec

2016-07-28 15:18:42:093, 2016-07-28 15:18:54:607, 0, 2400, 200, 114.44,
9.1450, 50000, 3995.5250

---------------

Staging:

kafka-producer-perf-test.sh --broker-list
kafkaStaging-1:9092,kafkaStaging-2:9092 --topics perf_test --message-size
2400 --messages 50000 --batch-size 200 --request-num-acks 1

start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec

2016-07-28 14:54:26:310, 2016-07-28 14:56:55:402, 0, 2400, 200, 114.44,
0.7676, 50000, 335.3634

---------------

That is a one magnitude difference in terms of nMsg/sec. Both clusters are
balanced. Does these numbers look right? The throughput on the staging
cluster looks quite low.

Thanks,

David