Getting best throughput without increasing latency with new producer/multiple threads producing

["Navneet Gupta (Tech - BLR)" <na...@flipkart.com>]

Hi,

I wanted to understand what kind of config settings will help me get the
best possible throughput with new producer API without increasing latency.

I tried to run the new producer shared between couple of threads and
latency seems to be increasing by increasing no. of threads. Ideally I
would like to keep latency between 3-5 ms and get as much throughput as
possible per client.

With 4 threads each producing 20k messages, I saw avg. latency to be around
2-3 ms. With 8 threads keeping all other things same, the avg. latency
increases to 4-6 ms. With 16 threads, its about 7-8 ms.

Apart from mandatory config, I am using pretty much standard configs with
just one change of using batch.size as 2048 currently (msg size is ~100
bytes).

-- 
Thanks & Regards,
Navneet Gupta