Kafka performance numbers on EC2

[Evan Chan <ev...@ooyala.com>]

Hi folks,

I did a performance test on EC2 using one Kafka box on an m1.large, and
separate m1.large boxes for the producer-perf-shell and the
consumer-perf-shell.
I'm using Kafka 0.6 with no mirroring.
With the message size set to 1000 bytes, here is what I found:

   - Writing Kafka messages to /mnt (EBS), we can write 21,190 messages per
   second.
   - If we write Kafka messages to /tmp (local disk), it's only slightly
   faster at 21,944 messages / sec
   - The number of partitions doesn't seem to have a bearing on
   write/consumer performance

It's pretty interesting that writing to /mnt or EBS doesn't seem to really
hurt the bandwidth.

I also found that the kafka-consumer-perf-test seems to misreport the
"nMsgs/sec".  It goes up from a small number at the beginning of the test
to a large number, which tells me that there is some elapsed / original
timestamp that was set waaay before the first message was consumed.  Maybe
this is fixed in 0.7?

I'd be interested to compare notes with others who have deployed Kafka to
EC2.

thanks,
Evan
-- 
--
*Evan Chan*
Senior Software Engineer |
ev@ooyala.com | (650) 996-4600
www.ooyala.com | blog <http://www.ooyala.com/blog> |
@ooyala<http://www.twitter.com/ooyala>

Re: Kafka performance numbers on EC2

[Jun Rao <ju...@gmail.com>]

Evan,

Thanks for sharing this. Not too sure about the difference btw EBS and
local disk. EBS could be caching writes in non-volatile memory. Therefore,
you only pay network cost, but not disk I/O cost.

We have fixed the reporting problem in performance test in 0.7. I recommend
that you try the 0.7 release.

Jun

On Thu, Feb 2, 2012 at 4:04 PM, Evan Chan <ev...@ooyala.com> wrote:

> Hi folks,
>
> I did a performance test on EC2 using one Kafka box on an m1.large, and
> separate m1.large boxes for the producer-perf-shell and the
> consumer-perf-shell.
> I'm using Kafka 0.6 with no mirroring.
> With the message size set to 1000 bytes, here is what I found:
>
>   - Writing Kafka messages to /mnt (EBS), we can write 21,190 messages per
>   second.
>   - If we write Kafka messages to /tmp (local disk), it's only slightly
>   faster at 21,944 messages / sec
>   - The number of partitions doesn't seem to have a bearing on
>   write/consumer performance
>
> It's pretty interesting that writing to /mnt or EBS doesn't seem to really
> hurt the bandwidth.
>
> I also found that the kafka-consumer-perf-test seems to misreport the
> "nMsgs/sec".  It goes up from a small number at the beginning of the test
> to a large number, which tells me that there is some elapsed / original
> timestamp that was set waaay before the first message was consumed.  Maybe
> this is fixed in 0.7?
>
> I'd be interested to compare notes with others who have deployed Kafka to
> EC2.
>
> thanks,
> Evan
> --
> --
> *Evan Chan*
> Senior Software Engineer |
> ev@ooyala.com | (650) 996-4600
> www.ooyala.com | blog <http://www.ooyala.com/blog> |
> @ooyala<http://www.twitter.com/ooyala>
>