You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by kant kodali <ka...@gmail.com> on 2016/09/15 09:00:03 UTC
What is the fair setup of Kafka to be comparable with NATS or NSQ?
with Kafka I tried it with 10 messages with single broker and only one partiton
that looked instantaneous and ~5K messages/sec for the data size of 1KB
I tried it with 1000 messages that looked instantaneous as well ~5K messages/sec
for the data size of 1KBI tried it with 10K messages with single broker and only
one partiton things started to go down ~1K messages/sec for the data size of
1KB
having only one partition on a single broker is a bad? My goal is to run some
basic benchmarks on NATS & NSQ & KAFKA
I have the same environment for all three (NATS & NSQ & KAFKA)
a broker on Machine 1producer on Machine 2Consumer on Machine 3
with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws instance.
I have pushed 300K messages with NATS and it was able to handle easily and
receive throughput was 5K messages/secI have pushed 300K messages and NSQ and
receive throughput was 2K messages/secI am unable to push 300K messages with
Kafka with the above configuration and environmentso at this point my biggest
question is what is the fair setup for Kafka so its comparable with NATS and
NSQ?
kant
Re: What is the fair setup of Kafka to be comparable with NATS or NSQ?
Posted by Ben Davison <be...@7digital.com>.
Yup, I suspect the client library.
On Thu, Sep 15, 2016 at 10:28 AM, kant kodali <ka...@gmail.com> wrote:
> Hi Ben,
> I can give that a try but can you tell me the suspicion or motivation
> behind it?
> other words you think single partition and single broker should be
> comparable to
> the setup I had with NATS and NSQ except you suspect the client library or
> something?
> Thanks,Kant
>
>
>
>
>
>
> On Thu, Sep 15, 2016 2:16 AM, Ben Davison ben.davison@7digital.com
> wrote:
>
> Hi Kant,
>
>
>
>
> I was following the other thread, can you try using a different
>
> benchmarking client for a test.
>
>
>
>
> https://grey-boundary.io/load-testing-apache-kafka-on-aws/
>
>
>
>
> Ben
>
>
>
>
> On Thursday, 15 September 2016, kant kodali <ka...@gmail.com> wrote:
>
>
>
>
> with Kafka I tried it with 10 messages with single broker and only one
>>
>
> partiton
>>
>
> that looked instantaneous and ~5K messages/sec for the data size of 1KB
>>
>
> I tried it with 1000 messages that looked instantaneous as well ~5K
>>
>
> messages/sec
>>
>
> for the data size of 1KBI tried it with 10K messages with single broker
>>
>
> and only
>>
>
> one partiton things started to go down ~1K messages/sec for the data size
>>
>
> of
>>
>
> 1KB
>>
>
> having only one partition on a single broker is a bad? My goal is to run
>>
>
> some
>>
>
> basic benchmarks on NATS & NSQ & KAFKA
>>
>
> I have the same environment for all three (NATS & NSQ & KAFKA)
>>
>
> a broker on Machine 1producer on Machine 2Consumer on Machine 3
>>
>
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
>>
>
> instance.
>>
>
> I have pushed 300K messages with NATS and it was able to handle easily and
>>
>
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
>>
>
> and
>>
>
> receive throughput was 2K messages/secI am unable to push 300K messages
>>
>
> with
>>
>
> Kafka with the above configuration and environmentso at this point my
>>
>
> biggest
>>
>
> question is what is the fair setup for Kafka so its comparable with NATS
>>
>
> and
>>
>
> NSQ?
>>
>
> kant
>>
>
>
>
>
> --
>
>
>
>
>
>
>
> This email, including attachments, is private and confidential. If you
> have
> received this email in error please notify the sender and delete it from
> your system. Emails are not secure and may contain viruses. No liability
> can be accepted for viruses that might be transferred by this email or any
> attachment. Any unauthorised copying of this message or unauthorised
> distribution and publication of the information contained herein are
> prohibited.
>
>
>
>
> 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
>
> Registered in England and Wales. Registered No. 04843573.
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
Re: What is the fair setup of Kafka to be comparable with NATS or
NSQ?
Posted by kant kodali <ka...@gmail.com>.
Hi Ben,
Also the article that you pointed out clearly shows the setup had multiple
partitions and multiple workers and so on..that's whyat this point my biggest
question is what is the fair setup for Kafka so its comparable with NATS and
NSQ? and since you suspect the client Library I can give that a go..but can you
please confirm that one partition on one broker should be able to handle 300K
messages of 1KB data size for each message?
Thanks,kant
On Thu, Sep 15, 2016 2:28 AM, kant kodali kanth909@gmail.com
wrote:
Hi Ben,
I can give that a try but can you tell me the suspicion or motivation behind it?
other words you think single partition and single broker should be comparable to
the setup I had with NATS and NSQ except you suspect the client library or
something?
Thanks,Kant
On Thu, Sep 15, 2016 2:16 AM, Ben Davison ben.davison@7digital.com
wrote:
Hi Kant,
I was following the other thread, can you try using a different
benchmarking client for a test.
https://grey-boundary.io/load-testing-apache-kafka-on-aws/
Ben
On Thursday, 15 September 2016, kant kodali <ka...@gmail.com> wrote:
> with Kafka I tried it with 10 messages with single broker and only one
> partiton
> that looked instantaneous and ~5K messages/sec for the data size of 1KB
> I tried it with 1000 messages that looked instantaneous as well ~5K
> messages/sec
> for the data size of 1KBI tried it with 10K messages with single broker
> and only
> one partiton things started to go down ~1K messages/sec for the data size
> of
> 1KB
> having only one partition on a single broker is a bad? My goal is to run
> some
> basic benchmarks on NATS & NSQ & KAFKA
> I have the same environment for all three (NATS & NSQ & KAFKA)
> a broker on Machine 1producer on Machine 2Consumer on Machine 3
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
> instance.
> I have pushed 300K messages with NATS and it was able to handle easily and
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
> and
> receive throughput was 2K messages/secI am unable to push 300K messages
> with
> Kafka with the above configuration and environmentso at this point my
> biggest
> question is what is the fair setup for Kafka so its comparable with NATS
> and
> NSQ?
> kant
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
Re: What is the fair setup of Kafka to be comparable with NATS or
NSQ?
Posted by kant kodali <ka...@gmail.com>.
Hi Ben,
I can give that a try but can you tell me the suspicion or motivation behind it?
other words you think single partition and single broker should be comparable to
the setup I had with NATS and NSQ except you suspect the client library or
something?
Thanks,Kant
On Thu, Sep 15, 2016 2:16 AM, Ben Davison ben.davison@7digital.com
wrote:
Hi Kant,
I was following the other thread, can you try using a different
benchmarking client for a test.
https://grey-boundary.io/load-testing-apache-kafka-on-aws/
Ben
On Thursday, 15 September 2016, kant kodali <ka...@gmail.com> wrote:
> with Kafka I tried it with 10 messages with single broker and only one
> partiton
> that looked instantaneous and ~5K messages/sec for the data size of 1KB
> I tried it with 1000 messages that looked instantaneous as well ~5K
> messages/sec
> for the data size of 1KBI tried it with 10K messages with single broker
> and only
> one partiton things started to go down ~1K messages/sec for the data size
> of
> 1KB
> having only one partition on a single broker is a bad? My goal is to run
> some
> basic benchmarks on NATS & NSQ & KAFKA
> I have the same environment for all three (NATS & NSQ & KAFKA)
> a broker on Machine 1producer on Machine 2Consumer on Machine 3
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
> instance.
> I have pushed 300K messages with NATS and it was able to handle easily and
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
> and
> receive throughput was 2K messages/secI am unable to push 300K messages
> with
> Kafka with the above configuration and environmentso at this point my
> biggest
> question is what is the fair setup for Kafka so its comparable with NATS
> and
> NSQ?
> kant
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.
Re: What is the fair setup of Kafka to be comparable with NATS or NSQ?
Posted by Ben Davison <be...@7digital.com>.
Hi Kant,
I was following the other thread, can you try using a different
benchmarking client for a test.
https://grey-boundary.io/load-testing-apache-kafka-on-aws/
Ben
On Thursday, 15 September 2016, kant kodali <ka...@gmail.com> wrote:
> with Kafka I tried it with 10 messages with single broker and only one
> partiton
> that looked instantaneous and ~5K messages/sec for the data size of 1KB
> I tried it with 1000 messages that looked instantaneous as well ~5K
> messages/sec
> for the data size of 1KBI tried it with 10K messages with single broker
> and only
> one partiton things started to go down ~1K messages/sec for the data size
> of
> 1KB
> having only one partition on a single broker is a bad? My goal is to run
> some
> basic benchmarks on NATS & NSQ & KAFKA
> I have the same environment for all three (NATS & NSQ & KAFKA)
> a broker on Machine 1producer on Machine 2Consumer on Machine 3
> with a data size of 1KB (so each message is 1KB ) and m4.xlarge aws
> instance.
> I have pushed 300K messages with NATS and it was able to handle easily and
> receive throughput was 5K messages/secI have pushed 300K messages and NSQ
> and
> receive throughput was 2K messages/secI am unable to push 300K messages
> with
> Kafka with the above configuration and environmentso at this point my
> biggest
> question is what is the fair setup for Kafka so its comparable with NATS
> and
> NSQ?
> kant
--
This email, including attachments, is private and confidential. If you have
received this email in error please notify the sender and delete it from
your system. Emails are not secure and may contain viruses. No liability
can be accepted for viruses that might be transferred by this email or any
attachment. Any unauthorised copying of this message or unauthorised
distribution and publication of the information contained herein are
prohibited.
7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB.
Registered in England and Wales. Registered No. 04843573.