You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Poorna Chandra Tejashvi Reddy <pc...@gmail.com> on 2015/09/02 23:03:12 UTC
Re: Kafka New Consumer Performance Test ThroughPut Degradation
Hi,
I have checked out the latest code out of https://github.com/apache/kafka based
on commit id e582447adb4708731aff74aa294e7ce2b30b0a41. Looks like the
performance test on the new-consumer is broken.
in/kafka-consumer-perf-test.sh --zookeeper zkip:2181 --broker-list
brokerIp:9092 --topic test --messages 50000 --new-consumer
The test does not return any response. Is this expected and is there a
better way to test the new-consumer.
Thanks,
-Poorna
On Thu, Aug 27, 2015 at 2:25 PM, Poorna Chandra Tejashvi Reddy <
pctreddy@gmail.com> wrote:
> Hi,
>
> We have built the latest kafka from https://github.com/apache/kafka based
> on this commit id 436b7ddc386eb688ba0f12836710f5e4bcaa06c8 .
> We ran the performance test on a 3 node kafka cluster. There is a huge
> throughput degradation using the new-consumer compared to the regular
> consumer. Below are the numbers that explain the same.
>
> bin/kafka-consumer-perf-test.sh --zookeeper zkIp:2181 --broker-list
> brokerIp:9092 --topics test --messages 5000000 : gives a throughput of 693 K
>
> bin/kafka-consumer-perf-test.sh --zookeeper zkIp:2181 --broker-list
> brokerIp:9092 --topics test --messages 5000000 --new-consumer : gives a
> throughput of 51k
>
> The whole set up is based on ec2, Kafka brokers running on r3.2x large.
>
> Are you guys aware of this performance degradation , do you have a JIRA
> for this, which can be used to track the resolution.
>
>
> Thanks,
>
> -Poorna
>
Re: Kafka New Consumer Performance Test ThroughPut Degradation
Posted by Poorna Chandra Tejashvi Reddy <pc...@gmail.com>.
Hi Jason,
Thanks for the update, I will be watching KAFKA-2489.
Thanks,
-Poorna
On Wed, Sep 2, 2015 at 5:31 PM, Jason Gustafson <ja...@confluent.io> wrote:
> Hi Poorna,
>
> We've been looking at the same issue as part of KAFKA-2489. I suspect the
> initial rebalance is causing the test to timeout, but we're still
> investigating.
>
> Thanks,
> Jason
>
> On Wed, Sep 2, 2015 at 2:03 PM, Poorna Chandra Tejashvi Reddy <
> pctreddy@gmail.com> wrote:
>
> > Hi,
> >
> > I have checked out the latest code out of
> https://github.com/apache/kafka
> > based
> > on commit id e582447adb4708731aff74aa294e7ce2b30b0a41. Looks like the
> > performance test on the new-consumer is broken.
> >
> > in/kafka-consumer-perf-test.sh --zookeeper zkip:2181 --broker-list
> > brokerIp:9092 --topic test --messages 50000 --new-consumer
> >
> > The test does not return any response. Is this expected and is there a
> > better way to test the new-consumer.
> >
> >
> > Thanks,
> >
> > -Poorna
> >
> >
> > On Thu, Aug 27, 2015 at 2:25 PM, Poorna Chandra Tejashvi Reddy <
> > pctreddy@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > We have built the latest kafka from https://github.com/apache/kafka
> > based
> > > on this commit id 436b7ddc386eb688ba0f12836710f5e4bcaa06c8 .
> > > We ran the performance test on a 3 node kafka cluster. There is a huge
> > > throughput degradation using the new-consumer compared to the regular
> > > consumer. Below are the numbers that explain the same.
> > >
> > > bin/kafka-consumer-perf-test.sh --zookeeper zkIp:2181 --broker-list
> > > brokerIp:9092 --topics test --messages 5000000 : gives a throughput of
> > 693 K
> > >
> > > bin/kafka-consumer-perf-test.sh --zookeeper zkIp:2181 --broker-list
> > > brokerIp:9092 --topics test --messages 5000000 --new-consumer : gives a
> > > throughput of 51k
> > >
> > > The whole set up is based on ec2, Kafka brokers running on r3.2x large.
> > >
> > > Are you guys aware of this performance degradation , do you have a JIRA
> > > for this, which can be used to track the resolution.
> > >
> > >
> > > Thanks,
> > >
> > > -Poorna
> > >
> >
>
Re: Kafka New Consumer Performance Test ThroughPut Degradation
Posted by Jason Gustafson <ja...@confluent.io>.
Hi Poorna,
We've been looking at the same issue as part of KAFKA-2489. I suspect the
initial rebalance is causing the test to timeout, but we're still
investigating.
Thanks,
Jason
On Wed, Sep 2, 2015 at 2:03 PM, Poorna Chandra Tejashvi Reddy <
pctreddy@gmail.com> wrote:
> Hi,
>
> I have checked out the latest code out of https://github.com/apache/kafka
> based
> on commit id e582447adb4708731aff74aa294e7ce2b30b0a41. Looks like the
> performance test on the new-consumer is broken.
>
> in/kafka-consumer-perf-test.sh --zookeeper zkip:2181 --broker-list
> brokerIp:9092 --topic test --messages 50000 --new-consumer
>
> The test does not return any response. Is this expected and is there a
> better way to test the new-consumer.
>
>
> Thanks,
>
> -Poorna
>
>
> On Thu, Aug 27, 2015 at 2:25 PM, Poorna Chandra Tejashvi Reddy <
> pctreddy@gmail.com> wrote:
>
> > Hi,
> >
> > We have built the latest kafka from https://github.com/apache/kafka
> based
> > on this commit id 436b7ddc386eb688ba0f12836710f5e4bcaa06c8 .
> > We ran the performance test on a 3 node kafka cluster. There is a huge
> > throughput degradation using the new-consumer compared to the regular
> > consumer. Below are the numbers that explain the same.
> >
> > bin/kafka-consumer-perf-test.sh --zookeeper zkIp:2181 --broker-list
> > brokerIp:9092 --topics test --messages 5000000 : gives a throughput of
> 693 K
> >
> > bin/kafka-consumer-perf-test.sh --zookeeper zkIp:2181 --broker-list
> > brokerIp:9092 --topics test --messages 5000000 --new-consumer : gives a
> > throughput of 51k
> >
> > The whole set up is based on ec2, Kafka brokers running on r3.2x large.
> >
> > Are you guys aware of this performance degradation , do you have a JIRA
> > for this, which can be used to track the resolution.
> >
> >
> > Thanks,
> >
> > -Poorna
> >
>