You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chen Song <ch...@gmail.com> on 2014/07/08 23:22:12 UTC
producer performance
While testing kafka producer performance, I found 2 testing scripts.
1) performance testing script in kafka distribution
bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages
10000000 --topic test --threads 10 --message-size 100 --batch-size 10000
--compression-codec 1
2) performance testing script mentioned in
https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
bin/kafka-run-class.sh
org.apache.kafka.clients.tools.ProducerPerformance test6 50000000 100
-1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092
buffer.memory=67108864 batch.size=8196
based on org.apache.kafka.clients.producer.Producer.
On my local testing environment,
For 1), the loading is ~15M/s and 150,000 msgs/s for mesage size = 100, no
matter how I adjust batch size and number of loading threads.
For 2), it can load up to 90M/s and 1,000,000 records/s.
I am wondering what makes these 2 implementations perform so differently?
--
Chen Song
Re: producer performance
Posted by Guozhang Wang <wa...@gmail.com>.
It depends on your network and the producer CPU, usually Kafka producer
will likely to saturate the network bandwidth first before it used up the
CPU if no compression is used.
On Fri, Jul 11, 2014 at 1:19 PM, Chen Song <ch...@gmail.com> wrote:
> Thanks Guaozhang.
>
> I have a follow-up question.
>
> Say if I can push 1M/s messages (100 size) into 2 node cluster. Is it safe
> to say I should be able to produce more or less 2M/s messages into 4 node
> cluster?
>
>
> On Tue, Jul 8, 2014 at 6:00 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > The second script is using the new producer which allows sending multiple
> > in-flight requests whereas the first script use the current (old)
> producer
> > which sends only one request at a time to a certain broker.
> >
> > The new producer will be officially released in 0.8.2.
> >
> > Guozhang
> >
> >
> > On Tue, Jul 8, 2014 at 2:22 PM, Chen Song <ch...@gmail.com>
> wrote:
> >
> > > While testing kafka producer performance, I found 2 testing scripts.
> > >
> > > 1) performance testing script in kafka distribution
> > >
> > > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages
> > > 10000000 --topic test --threads 10 --message-size 100 --batch-size
> 10000
> > > --compression-codec 1
> > >
> > > 2) performance testing script mentioned in
> > >
> > >
> > >
> >
> https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
> > >
> > > bin/kafka-run-class.sh
> > > org.apache.kafka.clients.tools.ProducerPerformance test6 50000000 100
> > > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092
> > > buffer.memory=67108864 batch.size=8196
> > >
> > > based on org.apache.kafka.clients.producer.Producer.
> > >
> > >
> > > On my local testing environment,
> > >
> > > For 1), the loading is ~15M/s and 150,000 msgs/s for mesage size = 100,
> > no
> > > matter how I adjust batch size and number of loading threads.
> > > For 2), it can load up to 90M/s and 1,000,000 records/s.
> > >
> > > I am wondering what makes these 2 implementations perform so
> differently?
> > >
> > > --
> > > Chen Song
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> Chen Song
>
--
-- Guozhang
Re: producer performance
Posted by Chen Song <ch...@gmail.com>.
Thanks Guaozhang.
I have a follow-up question.
Say if I can push 1M/s messages (100 size) into 2 node cluster. Is it safe
to say I should be able to produce more or less 2M/s messages into 4 node
cluster?
On Tue, Jul 8, 2014 at 6:00 PM, Guozhang Wang <wa...@gmail.com> wrote:
> The second script is using the new producer which allows sending multiple
> in-flight requests whereas the first script use the current (old) producer
> which sends only one request at a time to a certain broker.
>
> The new producer will be officially released in 0.8.2.
>
> Guozhang
>
>
> On Tue, Jul 8, 2014 at 2:22 PM, Chen Song <ch...@gmail.com> wrote:
>
> > While testing kafka producer performance, I found 2 testing scripts.
> >
> > 1) performance testing script in kafka distribution
> >
> > bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages
> > 10000000 --topic test --threads 10 --message-size 100 --batch-size 10000
> > --compression-codec 1
> >
> > 2) performance testing script mentioned in
> >
> >
> >
> https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
> >
> > bin/kafka-run-class.sh
> > org.apache.kafka.clients.tools.ProducerPerformance test6 50000000 100
> > -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092
> > buffer.memory=67108864 batch.size=8196
> >
> > based on org.apache.kafka.clients.producer.Producer.
> >
> >
> > On my local testing environment,
> >
> > For 1), the loading is ~15M/s and 150,000 msgs/s for mesage size = 100,
> no
> > matter how I adjust batch size and number of loading threads.
> > For 2), it can load up to 90M/s and 1,000,000 records/s.
> >
> > I am wondering what makes these 2 implementations perform so differently?
> >
> > --
> > Chen Song
> >
>
>
>
> --
> -- Guozhang
>
--
Chen Song
Re: producer performance
Posted by Guozhang Wang <wa...@gmail.com>.
The second script is using the new producer which allows sending multiple
in-flight requests whereas the first script use the current (old) producer
which sends only one request at a time to a certain broker.
The new producer will be officially released in 0.8.2.
Guozhang
On Tue, Jul 8, 2014 at 2:22 PM, Chen Song <ch...@gmail.com> wrote:
> While testing kafka producer performance, I found 2 testing scripts.
>
> 1) performance testing script in kafka distribution
>
> bin/kafka-producer-perf-test.sh --broker-list localhost:9092 --messages
> 10000000 --topic test --threads 10 --message-size 100 --batch-size 10000
> --compression-codec 1
>
> 2) performance testing script mentioned in
>
>
> https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
>
> bin/kafka-run-class.sh
> org.apache.kafka.clients.tools.ProducerPerformance test6 50000000 100
> -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092
> buffer.memory=67108864 batch.size=8196
>
> based on org.apache.kafka.clients.producer.Producer.
>
>
> On my local testing environment,
>
> For 1), the loading is ~15M/s and 150,000 msgs/s for mesage size = 100, no
> matter how I adjust batch size and number of loading threads.
> For 2), it can load up to 90M/s and 1,000,000 records/s.
>
> I am wondering what makes these 2 implementations perform so differently?
>
> --
> Chen Song
>
--
-- Guozhang