You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Marc Labbe <mr...@gmail.com> on 2015/03/13 16:10:36 UTC

Kafka High CPU, 0.8.2.1 or openjdk?

Hi,

our cluster is deployed on AWS, we have brokers on r3.large instances, a
decent amount of topics+partitions (+600 partitions). We're not making that
many requests/sec, roughly 80 produce/sec and 240 fetch/sec (not counting
internal replication requests) and yet CPU hovers around 40%, which I
consider quite high given the nature of Kafka. I have worked on other
deployments not on AWS where we were getting much larger figures in
requests/sec, w/ much less CPU usage than that.

There are two things I consider trying to reduce that. The first one is
obviously using 0.8.2.1 though I am not sure how much the impact is. I also
found this thread from a while ago
http://mail-archives.apache.org/mod_mbox/kafka-users/201305.mbox/%3CEB51B84C-AD91-4A2F-B97D-5283EF079596@transpac.com%3E
about the use of the OpenJDK (which we're using, Ubuntu trusty's default)
vs Oracle JDK.

I am planning to do both anyways but I thought it'd be interesting to know
if anyone else has experienced that before. Is there any other tuning I
should think about?

thanks
marc

Re: Kafka High CPU, 0.8.2.1 or openjdk?

Posted by Marc Labbe <mr...@gmail.com>.

Thanks, I'll start with that before changing my deployment for oracle jdk.


On Fri, Mar 13, 2015 at 11:40 AM, Mark Reddy <ma...@gmail.com> wrote:

> Hi Marc,
>
> If you are seeing high CPU usages with a large number of partitions on
> 0.8.2 you should definitely upgrade to 0.8.2.1 as the following issue was
> fixed: https://issues.apache.org/jira/browse/KAFKA-1952
>
> Also see the 0.8.2.1 release notes for other fixes:
> https://archive.apache.org/dist/kafka/0.8.2.1/RELEASE_NOTES.html
>
>
> Mark
>
> On 13 March 2015 at 15:10, Marc Labbe <mr...@gmail.com> wrote:
>
> > Hi,
> >
> > our cluster is deployed on AWS, we have brokers on r3.large instances, a
> > decent amount of topics+partitions (+600 partitions). We're not making
> that
> > many requests/sec, roughly 80 produce/sec and 240 fetch/sec (not counting
> > internal replication requests) and yet CPU hovers around 40%, which I
> > consider quite high given the nature of Kafka. I have worked on other
> > deployments not on AWS where we were getting much larger figures in
> > requests/sec, w/ much less CPU usage than that.
> >
> > There are two things I consider trying to reduce that. The first one is
> > obviously using 0.8.2.1 though I am not sure how much the impact is. I
> also
> > found this thread from a while ago
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201305.mbox/%3CEB51B84C-AD91-4A2F-B97D-5283EF079596@transpac.com%3E
> > about the use of the OpenJDK (which we're using, Ubuntu trusty's default)
> > vs Oracle JDK.
> >
> > I am planning to do both anyways but I thought it'd be interesting to
> know
> > if anyone else has experienced that before. Is there any other tuning I
> > should think about?
> >
> > thanks
> > marc
> >
>

Re: Kafka High CPU, 0.8.2.1 or openjdk?

Posted by Mark Reddy <ma...@gmail.com>.

Hi Marc,

If you are seeing high CPU usages with a large number of partitions on
0.8.2 you should definitely upgrade to 0.8.2.1 as the following issue was
fixed: https://issues.apache.org/jira/browse/KAFKA-1952

Also see the 0.8.2.1 release notes for other fixes:
https://archive.apache.org/dist/kafka/0.8.2.1/RELEASE_NOTES.html


Mark

On 13 March 2015 at 15:10, Marc Labbe <mr...@gmail.com> wrote:

> Hi,
>
> our cluster is deployed on AWS, we have brokers on r3.large instances, a
> decent amount of topics+partitions (+600 partitions). We're not making that
> many requests/sec, roughly 80 produce/sec and 240 fetch/sec (not counting
> internal replication requests) and yet CPU hovers around 40%, which I
> consider quite high given the nature of Kafka. I have worked on other
> deployments not on AWS where we were getting much larger figures in
> requests/sec, w/ much less CPU usage than that.
>
> There are two things I consider trying to reduce that. The first one is
> obviously using 0.8.2.1 though I am not sure how much the impact is. I also
> found this thread from a while ago
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201305.mbox/%3CEB51B84C-AD91-4A2F-B97D-5283EF079596@transpac.com%3E
> about the use of the OpenJDK (which we're using, Ubuntu trusty's default)
> vs Oracle JDK.
>
> I am planning to do both anyways but I thought it'd be interesting to know
> if anyone else has experienced that before. Is there any other tuning I
> should think about?
>
> thanks
> marc
>