Kafka consumer : Maximum consumer rate.

[Mazen Ezzeddine <ma...@etu.univ-cotedazur.fr>]

I am interested in learning/deducing the maximum consumption rate of a Kafka consumer in my consumer group. Maximum consumption rate is the rate at which the consumer can not keep up with the message arrival rate, and hence the consumer will fall farther and farther behind and the message lag would accumulate.

On the consumer side, I can compute the maximum consumption rate as 1/(message waiting time), where message waiting time is the time required to pull the message from the broker and process it by the consumer in the poll loop.

Would that strategy return an ACCURATE measure of the maximum consumption rate of a consumer. Any other suggestion for the computation of the maximum consumption rate using the consumer API?

On the other hand, say I have a controller process with a Kafka admin client API, would the below logic return accurately the maximum consumption rate (any unseen boundary cases). In the logic, At and Rt are the arrival and consumer rate at time t, and at time t-1 for At-1 and Rt-1

if (At > At-1 && Rt = Rt-1) then Rt is the maximum consumption rate.


Any suggestion on the computation of the maximum consumption rate using the admin client API and not the consumer API?


Thank you.