generate specific throughput load

[Josh J <jo...@gmail.com>]

Hi,

Is there a way to generate a specified amount of throughput? I'm using the
Stats class here
<https://github.com/apache/kafka/blob/7130da90a9ee9e6fb4beb2a2a6ab05c06c9bfac4/clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java>
to
measure the throughput. Though I need to be able to precisely control the
amount of load. For example, 1000 records per second.

Thanks,
Josh

Re: generate specific throughput load

[Jiangjie Qin <jq...@linkedin.com.INVALID>]

There is this ProducerPerformance class coming with new java producer. You
can go to KAFKA_HOME/bin and use the following command:

./kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance
USAGE: java org.apache.kafka.clients.tools.ProducerPerformance topic_name
num_records record_size target_records_sec [prop_name=prop_value]*


In you need to specify bootstrap.servers in the property.

Jiangjie (Becket) Qin


On 2/25/15, 11:28 AM, "Magnus Edenhill" <ma...@edenhill.se> wrote:

>Hi,
>
>the rdkafka_performance tool from librdkafka's examples [1] lets you do
>this with something like:
>rdkafka_performance -P -b <broker> -t <topic> [-p <partition>] -r <msgs/s>
>-s <msgsize>
>
>Thats the producer side, if you want performance measurements on the
>consumer side as well you do:
>rdkafka_performance -C -b <broker> -t <topic> -p <partition> -o
><start_offset|beginning|end>
>
>There is an end-to-end latency measurement mode too where you run both
>producer and consumer simultaneously with the '-l' option.
>(this mode typically requires some config tweaking to get optimal results,
>e.g. minimize batch wait timing)
>
>Regards,
>Magnus
>
>[1]: https://github.com/edenhill/librdkafka
>
>2015-02-25 19:46 GMT+01:00 Josh J <jo...@gmail.com>:
>
>> Hi,
>>
>> Is there a way to generate a specified amount of throughput? I'm using
>>the
>> Stats class here
>> <
>> 
>>https://github.com/apache/kafka/blob/7130da90a9ee9e6fb4beb2a2a6ab05c06c9b
>>fac4/clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerform
>>ance.java
>> >
>> to
>> measure the throughput. Though I need to be able to precisely control
>>the
>> amount of load. For example, 1000 records per second.
>>
>> Thanks,
>> Josh
>>

Re: generate specific throughput load

[Magnus Edenhill <ma...@edenhill.se>]

Hi,

the rdkafka_performance tool from librdkafka's examples [1] lets you do
this with something like:
rdkafka_performance -P -b <broker> -t <topic> [-p <partition>] -r <msgs/s>
-s <msgsize>

Thats the producer side, if you want performance measurements on the
consumer side as well you do:
rdkafka_performance -C -b <broker> -t <topic> -p <partition> -o
<start_offset|beginning|end>

There is an end-to-end latency measurement mode too where you run both
producer and consumer simultaneously with the '-l' option.
(this mode typically requires some config tweaking to get optimal results,
e.g. minimize batch wait timing)

Regards,
Magnus

[1]: https://github.com/edenhill/librdkafka

2015-02-25 19:46 GMT+01:00 Josh J <jo...@gmail.com>:

> Hi,
>
> Is there a way to generate a specified amount of throughput? I'm using the
> Stats class here
> <
> https://github.com/apache/kafka/blob/7130da90a9ee9e6fb4beb2a2a6ab05c06c9bfac4/clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java
> >
> to
> measure the throughput. Though I need to be able to precisely control the
> amount of load. For example, 1000 records per second.
>
> Thanks,
> Josh
>