Producer throughput question

[Maung Than <ma...@apple.com>]

Hi All, 

What is the good way to measure the producer throughput? 

We are currently accumulating time take by each individual send in the producer and then dividing it with the total volume to calculate the producer throughput. We seems to be getting 20% more throughput than our network can support that is 1000 mbps. 

We also find compression (such as GZip) does not add more time to the send. Is the compression done in the send thread or in a separate thread for async mode?

async with batch size 1000
No of Partitions = 5
Replication factor = 3
No of brokers = 5; 
No of producers = 2; 

Thanks,
Maung

Re: Producer throughput question

[Joel Koshy <jj...@gmail.com>]

> 
> What is the good way to measure the producer throughput? 

The producer exposes a number of mbeans that you can poke to monitor
throughput (e.g., bytes-per-sec, messages-per-sec, etc.) and it does
so on an aggregate as well as per-topic basis.

> We also find compression (such as GZip) does not add more time to the send. Is the compression done in the send thread or in a separate thread for async mode?
> 

In the current producer implementation it is done in a separate thread
(i.e., separate from the calling thread). The send also happens in
that separate thread.

In the new producer implementation (under kafka.clients) the
compression happens in the calling (client thread) and the send
happens in a separate thread.

Joel