Kafka topic message consumer fetch response time checking

[David Luu <ma...@gmail.com>]

If one wanted to check response time for a consumer fetching a topic
message (on the client side), similar to checking an HTTP request's
response time for the web, what's the best approach to take?

I notice the kafka shell scripts if used for that have some startup
overhead if used to assess response times frequently.

Is there a speedier CLI alternative, or will one have to write a custom
script/program in one of the kafka language clients for less startup
overhead for frequent periodic checks?

Or is there a kafka server side metric that is sufficient to check for this
(no need for client side checking, just look at the server metrics)?

Re: Kafka topic message consumer fetch response time checking

[Ewen Cheslack-Postava <ew...@confluent.io>]

If you want the full round-trip latency, you need to measure that at the
client. The performance measurement tools should do this pretty accurately.
For example, if you just want to know how long it takes to produce a
message to Kafka and get an ack back, you can use the latency numbers
reported by ProducerPerformance, ensuring to use a low throughput to avoid
queuing (so you measure the latency rather than queuing delay). If you
wanted producer -> brokers -> consumer latency, use EndToEndLatency.

There is some startup cost, but they don't start timing anything until a
lot of the startup has already taken place and as long as you run a long
enough test, the impact of the initial startup cost shouldn't be noticeable.

-Ewen

On Mon, Oct 19, 2015 at 2:45 PM, David Luu <ma...@gmail.com> wrote:

> If one wanted to check response time for a consumer fetching a topic
> message (on the client side), similar to checking an HTTP request's
> response time for the web, what's the best approach to take?
>
> I notice the kafka shell scripts if used for that have some startup
> overhead if used to assess response times frequently.
>
> Is there a speedier CLI alternative, or will one have to write a custom
> script/program in one of the kafka language clients for less startup
> overhead for frequent periodic checks?
>
> Or is there a kafka server side metric that is sufficient to check for this
> (no need for client side checking, just look at the server metrics)?
>



-- 
Thanks,
Ewen