You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by James Brown <jb...@easypost.com> on 2019/12/27 18:07:11 UTC
Significant CPU utilization increase in 2.4.0
I just upgraded one of our test clusters from 2.3.1 to 2.4.0 and the system
CPU usage very noticeably increased (from approximately 35% of a CPU to
approximately 50% of a CPU). The cluster is very lightly loaded (around 150
messages/sec in aggregate). This CPU time increase is visible symmetrically
on all three nodes in the cluster.
The CPU consumption did not return to normal after I did the second restart
to bump the log and inter-broker protocol versions to 2.4, so I don't think
it has anything to do with down-converting to the 2.3 protocols.
No settings were changed, nor was anything about the JVM changed. There is
nothing interesting being written to the logs. There's no sign of any
instability (partitions aren't being reassigned, etc).
The best guess I have for the increased CPU usage is that the number of
garbage collections increased by approximately 30%, suggesting that
something is churning a lot more garbage inside Kafka. This is a small
cluster, so it's only got a 3GB heap allocated to Kafka on each node; we're
using G1GC with some light tuning and are on Java 8 if that helps.
Are these regressions expected? Should I expect them to be constant or
worse when we upgrade something with more load on it?
--
James Brown
Engineer
Re: Significant CPU utilization increase in 2.4.0
Posted by James Brown <jb...@easypost.com>.
I filed https://issues.apache.org/jira/browse/KAFKA-9339
On Fri, Dec 27, 2019 at 12:24 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi James,
>
> Thanks for the report. This is not expected. How many partitions per broker
> and what Java version? I suggest filing a JIRA and, if possible, attach a
> flight recorder profile.
>
> Ismael
>
>
> On Fri, Dec 27, 2019, 7:07 PM James Brown <jb...@easypost.com> wrote:
>
> > I just upgraded one of our test clusters from 2.3.1 to 2.4.0 and the
> system
> > CPU usage very noticeably increased (from approximately 35% of a CPU to
> > approximately 50% of a CPU). The cluster is very lightly loaded (around
> 150
> > messages/sec in aggregate). This CPU time increase is visible
> symmetrically
> > on all three nodes in the cluster.
> >
> > The CPU consumption did not return to normal after I did the second
> restart
> > to bump the log and inter-broker protocol versions to 2.4, so I don't
> think
> > it has anything to do with down-converting to the 2.3 protocols.
> >
> > No settings were changed, nor was anything about the JVM changed. There
> is
> > nothing interesting being written to the logs. There's no sign of any
> > instability (partitions aren't being reassigned, etc).
> >
> > The best guess I have for the increased CPU usage is that the number of
> > garbage collections increased by approximately 30%, suggesting that
> > something is churning a lot more garbage inside Kafka. This is a small
> > cluster, so it's only got a 3GB heap allocated to Kafka on each node;
> we're
> > using G1GC with some light tuning and are on Java 8 if that helps.
> >
> > Are these regressions expected? Should I expect them to be constant or
> > worse when we upgrade something with more load on it?
> > --
> > James Brown
> > Engineer
> >
>
--
James Brown
Engineer
Re: Significant CPU utilization increase in 2.4.0
Posted by Ismael Juma <is...@juma.me.uk>.
Hi James,
Thanks for the report. This is not expected. How many partitions per broker
and what Java version? I suggest filing a JIRA and, if possible, attach a
flight recorder profile.
Ismael
On Fri, Dec 27, 2019, 7:07 PM James Brown <jb...@easypost.com> wrote:
> I just upgraded one of our test clusters from 2.3.1 to 2.4.0 and the system
> CPU usage very noticeably increased (from approximately 35% of a CPU to
> approximately 50% of a CPU). The cluster is very lightly loaded (around 150
> messages/sec in aggregate). This CPU time increase is visible symmetrically
> on all three nodes in the cluster.
>
> The CPU consumption did not return to normal after I did the second restart
> to bump the log and inter-broker protocol versions to 2.4, so I don't think
> it has anything to do with down-converting to the 2.3 protocols.
>
> No settings were changed, nor was anything about the JVM changed. There is
> nothing interesting being written to the logs. There's no sign of any
> instability (partitions aren't being reassigned, etc).
>
> The best guess I have for the increased CPU usage is that the number of
> garbage collections increased by approximately 30%, suggesting that
> something is churning a lot more garbage inside Kafka. This is a small
> cluster, so it's only got a 3GB heap allocated to Kafka on each node; we're
> using G1GC with some light tuning and are on Java 8 if that helps.
>
> Are these regressions expected? Should I expect them to be constant or
> worse when we upgrade something with more load on it?
> --
> James Brown
> Engineer
>