Kafka0.8 Performance

[Rakesh Nair <ra...@gmail.com>]

I was running  couple of performance tests using kafka 0.8. I needed to get
the perf numbers for sync producers with ack's. I am getting data transfer
speeds of around 36MB/sec. But the CPU util is well below 50% and the
IOstat are also normal. Even if i push more data using larger number of
threads, am still getting data transfer speeds of around 30MB/sec.

What should i do to get better transfer speeds, since obviously the cpu and
io dont seem to be a bottleneck.
Could you tell me what i am doing wrong or are there any configurations
that i need to alter.

Here is one of the configurations that i had used:

no. of  processors available = 24

server.properties configurations:
 network.threads=20
 io.threads=20
 partitions = 20
 log.flush.interval=500
 log.defualt.flush.interval=3000
 log.default.flush.scheduler.interval.ms=3000

Perf command : ./kafka-producer-perf-test.sh --topic perf_0 --broker-list
localhost:9092 --message-size 1024 --request-num-acks 1 --batch-size 1
 --threads 20  --messages 1000000
Transfer speed : ~31MB/sec
-- 
Regards
Rakesh Nair

Re: Kafka0.8 Performance

[Jun Rao <ju...@gmail.com>]

Could you try increasing the batch size and use async produce? Thanks,

Jun

On Tue, Aug 28, 2012 at 11:39 AM, Rakesh Nair <ra...@gmail.com> wrote:

> I was running  couple of performance tests using kafka 0.8. I needed to get
> the perf numbers for sync producers with ack's. I am getting data transfer
> speeds of around 36MB/sec. But the CPU util is well below 50% and the
> IOstat are also normal. Even if i push more data using larger number of
> threads, am still getting data transfer speeds of around 30MB/sec.
>
> What should i do to get better transfer speeds, since obviously the cpu and
> io dont seem to be a bottleneck.
> Could you tell me what i am doing wrong or are there any configurations
> that i need to alter.
>
> Here is one of the configurations that i had used:
>
> no. of  processors available = 24
>
> server.properties configurations:
>  network.threads=20
>  io.threads=20
>  partitions = 20
>  log.flush.interval=500
>  log.defualt.flush.interval=3000
>  log.default.flush.scheduler.interval.ms=3000
>
> Perf command : ./kafka-producer-perf-test.sh --topic perf_0 --broker-list
> localhost:9092 --message-size 1024 --request-num-acks 1 --batch-size 1
>  --threads 20  --messages 1000000
> Transfer speed : ~31MB/sec
> --
> Regards
> Rakesh Nair
>