You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Lu Xuechao <lu...@gmail.com> on 2013/10/21 21:01:23 UTC

Please help find the reason behind performance test results.

Hi,

We observed below correlation between kafka configuration and performance
data:

1. producer insertion rate drops as compression enabled, especially gzip;
2. when the producer batch size is below 200, we got high CPU/network IO on
brokers, got high network IO on producers/consumers;

What's the reasons behind?

thanks.

Re: Please help find the reason behind performance test results.

Posted by Lu Xuechao <lu...@gmail.com>.

Yes, I am using 0.8.
Network IO means bytes transferred.

Thanks for reply.


On Mon, Oct 21, 2013 at 2:24 PM, Neha Narkhede <ne...@gmail.com>wrote:

> Are these for Kafka 08?
>
> For #2 above, when you say high network I/O, do you mean number of packets
> transferred or size in bytes transferred?
>
> Thanks,
> Neha
>
>
> On Mon, Oct 21, 2013 at 12:01 PM, Lu Xuechao <lu...@gmail.com> wrote:
>
> > Hi,
> >
> > We observed below correlation between kafka configuration and performance
> > data:
> >
> > 1. producer insertion rate drops as compression enabled, especially gzip;
> > 2. when the producer batch size is below 200, we got high CPU/network IO
> on
> > brokers, got high network IO on producers/consumers;
> >
> > What's the reasons behind?
> >
> > thanks.
> >
>

Re: Please help find the reason behind performance test results.

Posted by Neha Narkhede <ne...@gmail.com>.

Are these for Kafka 08?

For #2 above, when you say high network I/O, do you mean number of packets
transferred or size in bytes transferred?

Thanks,
Neha

On Mon, Oct 21, 2013 at 12:01 PM, Lu Xuechao <lu...@gmail.com> wrote:

> Hi,
>
> We observed below correlation between kafka configuration and performance
> data:
>
> 1. producer insertion rate drops as compression enabled, especially gzip;
> 2. when the producer batch size is below 200, we got high CPU/network IO on
> brokers, got high network IO on producers/consumers;
>
> What's the reasons behind?
>
> thanks.
>

Re: Please help find the reason behind performance test results.

Posted by Jun Rao <ju...@gmail.com>.

If the data is compressed, the broker has to recompress the messages in
order to assign offsets. So, there is some CPU overhead. However, it
shouldn't be too high. How high of the CPU load did you observe?

Thanks,

Jun

On Mon, Oct 21, 2013 at 12:01 PM, Lu Xuechao <lu...@gmail.com> wrote:

> Hi,
>
> We observed below correlation between kafka configuration and performance
> data:
>
> 1. producer insertion rate drops as compression enabled, especially gzip;
> 2. when the producer batch size is below 200, we got high CPU/network IO on
> brokers, got high network IO on producers/consumers;
>
> What's the reasons behind?
>
> thanks.
>