Kafka Streams Latency

[Adrienne Kole <ad...@gmail.com>]

Hi,

 How can I measure the latency and throughput in Kafka Streams?

Cheers
Adrienne

Re: Kafka Streams Latency

[Michael Noll <mi...@confluent.io>]

The SimpleBenchmark included in Apache Kafka [1] is, as Eno mentioned,
quite rudimentary.  It does some simple benchmark runs of Kafka's
standard/native producer and consumer clients, and then some Kafka Streams
specific ones.  So you can compare somewhat between the native clients and
Kafka Streams (which uses the producer and consumer clients behind the
scenes).

By default, the SimpleBenchmark assumes that you have a local ZK instance
running on port 2181 plus a local Kafka broker running on port 9092.  But
note that, in practice, ZooKeeper, Kafka (broker/brokers), and a Kafka
Streams application are usually on different machines, so throughput
numbers from an environment where you co-located these three components on
a single machine (as is assumed by default by SimpleBenchmark) needs to be
taken with a grain of salt.

One way to run SimpleBenchmark is as follows.

- Checkout, build, and locally install Kafka's trunk version.

    $ git clone https://github.com/apache/kafka.git
    $ cd kafka

    # May or may not be needed: bootstrap gradle wrapper (to give you
`gradlew`)
    $ gradle

    # Note: perhaps e.g. `./gradlew jarAll` would be sufficient, I can't
remember off the top of my head
    $ ./gradlew installAll


- Run a local Zookeeper instance (from the cloned repo top-level dir):

    $ bin/zookeeper-server-start.sh config/zookeeper.properties

- Then, in another terminal, run a local Kafka broker (from the cloned repo
top-level dir):

    $ bin/kafka-server-start.sh config/server.properties

- Then, in another terminal or from within your IDE, run SimpleBenchmark.


Hope this helps!
Michael



[1]
https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/perf/SimpleBenchmark.java


On Fri, Jul 22, 2016 at 4:30 PM, Eno Thereska <en...@gmail.com>
wrote:

> Hi Adrienne,
>
> Also you might want to have a look at the SimpleBenchmark.java file
> included with Kafka Streams (org.apache.kafka.streams.perf.
> SimpleBenchmark). It does some simple measurements of consumer, producer
> and Kafka Streams throughput.
>
> Thanks
> Eno
>
> > On 22 Jul 2016, at 07:21, David Garcia <da...@spiceworks.com> wrote:
> >
> > You should probably just put reporting in your app.  Dropwizard,
> logs…etc.  You can also look at Kafka JMX consumer metrics (assuming you
> don’t have too many consumers).
> >
> > -David
> >
> > On 7/22/16, 9:13 AM, "Adrienne Kole" <ad...@gmail.com> wrote:
> >
> >    Hi,
> >
> >     How can I measure the latency and throughput in Kafka Streams?
> >
> >    Cheers
> >    Adrienne
> >
> >
>
>


-- 

*Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
Apache Kafka and Confluent Platform: www.confluent.io/download
<http://www.confluent.io/download>*

Re: Kafka Streams Latency

[Eno Thereska <en...@gmail.com>]

Hi Adrienne,

Also you might want to have a look at the SimpleBenchmark.java file included with Kafka Streams (org.apache.kafka.streams.perf. SimpleBenchmark). It does some simple measurements of consumer, producer and Kafka Streams throughput.

Thanks
Eno

> On 22 Jul 2016, at 07:21, David Garcia <da...@spiceworks.com> wrote:
> 
> You should probably just put reporting in your app.  Dropwizard, logs…etc.  You can also look at Kafka JMX consumer metrics (assuming you don’t have too many consumers).
> 
> -David
> 
> On 7/22/16, 9:13 AM, "Adrienne Kole" <ad...@gmail.com> wrote:
> 
>    Hi,
> 
>     How can I measure the latency and throughput in Kafka Streams?
> 
>    Cheers
>    Adrienne
> 
> 

Re: Kafka Streams Latency

[David Garcia <da...@spiceworks.com>]

You should probably just put reporting in your app.  Dropwizard, logs…etc.  You can also look at Kafka JMX consumer metrics (assuming you don’t have too many consumers).

-David

On 7/22/16, 9:13 AM, "Adrienne Kole" <ad...@gmail.com> wrote:

    Hi,
    
     How can I measure the latency and throughput in Kafka Streams?
    
    Cheers
    Adrienne