You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Anuja Pundlik (apundlik)" <ap...@cisco.com> on 2015/06/04 06:43:23 UTC
High CPU usage for idle kafka server
Hi,
I am using Kafka 0.8.2.1.
We have 1 zookeeper, 3 kafka brokers.
We have 9 topics, out of which 1 topic has 18 partitions, while another has 12 partitions. All other topics have 1 partition each.
We see that idle kafka brokers (not carrying any message) are using more than 50% of CPU. See top output below.
Is this a known issue?
Thanks
top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si, 0.4%st
Mem: 65974296k total, 22310524k used, 43663772k free, 112688k buffers
Swap: 0k total, 0k used, 0k free, 13382460k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
Re: High CPU usage for idle kafka server
Posted by "pundlik.anuja" <pu...@gmail.com>.
There are no messages being sent or received. The system is idle.
however, there seems to be some GC going on in the kafka broker and some
socket reads and writes. It is using approx 500MB of memory.
On Fri, Jun 5, 2015 at 1:44 PM, pundlik.anuja <pu...@gmail.com>
wrote:
> Hi Otis,
> How do I check garbage collection on kafka broker?
>
>
> On Thu, Jun 4, 2015 at 1:24 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
>> How's their garbage collection doing?
>>
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On Thu, Jun 4, 2015 at 12:43 AM, Anuja Pundlik (apundlik) <
>> apundlik@cisco.com> wrote:
>>
>> > Hi,
>> >
>> > I am using Kafka 0.8.2.1.
>> > We have 1 zookeeper, 3 kafka brokers.
>> > We have 9 topics, out of which 1 topic has 18 partitions, while another
>> > has 12 partitions. All other topics have 1 partition each.
>> >
>> > We see that idle kafka brokers (not carrying any message) are using more
>> > than 50% of CPU. See top output below.
>> >
>> > Is this a known issue?
>> >
>> >
>> > Thanks
>> >
>> >
>> >
>> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
>> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
>> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
>> > 0.4%st
>> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k buffers
>> > Swap: 0k total, 0k used, 0k free, 13382460k cached
>> >
>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
>> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
>> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
>> >
>>
>
>
Re: High CPU usage for idle kafka server
Posted by "pundlik.anuja" <pu...@gmail.com>.
Hi Otis,
How do I check garbage collection on kafka broker?
On Thu, Jun 4, 2015 at 1:24 PM, Otis Gospodnetic <otis.gospodnetic@gmail.com
> wrote:
> How's their garbage collection doing?
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Thu, Jun 4, 2015 at 12:43 AM, Anuja Pundlik (apundlik) <
> apundlik@cisco.com> wrote:
>
> > Hi,
> >
> > I am using Kafka 0.8.2.1.
> > We have 1 zookeeper, 3 kafka brokers.
> > We have 9 topics, out of which 1 topic has 18 partitions, while another
> > has 12 partitions. All other topics have 1 partition each.
> >
> > We see that idle kafka brokers (not carrying any message) are using more
> > than 50% of CPU. See top output below.
> >
> > Is this a known issue?
> >
> >
> > Thanks
> >
> >
> >
> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
> > 0.4%st
> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k buffers
> > Swap: 0k total, 0k used, 0k free, 13382460k cached
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
> >
>
Re: High CPU usage for idle kafka server
Posted by Otis Gospodnetic <ot...@gmail.com>.
How's their garbage collection doing?
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Thu, Jun 4, 2015 at 12:43 AM, Anuja Pundlik (apundlik) <
apundlik@cisco.com> wrote:
> Hi,
>
> I am using Kafka 0.8.2.1.
> We have 1 zookeeper, 3 kafka brokers.
> We have 9 topics, out of which 1 topic has 18 partitions, while another
> has 12 partitions. All other topics have 1 partition each.
>
> We see that idle kafka brokers (not carrying any message) are using more
> than 50% of CPU. See top output below.
>
> Is this a known issue?
>
>
> Thanks
>
>
>
> top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
> Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
> Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
> 0.4%st
> Mem: 65974296k total, 22310524k used, 43663772k free, 112688k buffers
> Swap: 0k total, 0k used, 0k free, 13382460k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
> 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
> 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
>
Re: High CPU usage for idle kafka server
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
I guess apache mailing list does not support attachments…
Maybe you can paste it some where and send the link?
Jiangjie (Becket) Qin
From: "pundlik.anuja" <pu...@gmail.com>>
Reply-To: "users@kafka.apache.org<ma...@kafka.apache.org>" <us...@kafka.apache.org>>
Date: Monday, June 8, 2015 at 1:53 PM
To: "users@kafka.apache.org<ma...@kafka.apache.org>" <us...@kafka.apache.org>>
Subject: Re: High CPU usage for idle kafka server
Hi Jiangjie,
Resending profiling screenshot for High CPU usage on Idle kafka broker.
Appreciate your help.
Thanks,
Anuja
On Mon, Jun 8, 2015 at 1:45 PM, pundlik.anuja <pu...@gmail.com>> wrote:
Yes, partly it seems to be due to log cleaner thread.
I changed log cleaning settings for each broker (server*.properties) file.
# The minimum age of a log file to be eligible for deletion
log.retention.hours=1
# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=536870912
# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms<http://log.retention.check.interval.ms>=10000 (****** This was 1 ms before)
# By default the log cleaner is disabled and the log retention policy will default to just delete segments after their retention expires.
# If log.cleaner.enable=true is set the cleaner will be enabled and individual logs can then be marked for log compaction.
log.cleaner.enable=false (***** this was true before)
Question:
Per my understanding, setting log.cleaner.enable=false will cause deletion of logs instead of compaction. Is that correct? Or will it cause the logs to keep on accumulating and filling up the disk space?
THanks,
Anuja
On Mon, Jun 8, 2015 at 1:24 PM, Jay Kreps <ja...@gmail.com>> wrote:
Could it also be that the log cleaner is running? This will definitely use
some CPU while the cleaning is occurring (it would attempt to use one cpu
per log cleaner thread you configure).
-Jay
On Mon, Jun 8, 2015 at 1:07 PM, Jiangjie Qin <jq...@linkedin.com.invalid>>
wrote:
> It seems the attachments are lost. But high CPU for ReplicaRetcherThread
> might be related to KAFKA-1461. Can you try to apply that patch and see if
> it solves problem?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> From: "pundlik.anuja" <pu...@gmail.com><mailto:
> pundlik.anuja@gmail.com<ma...@gmail.com>>>
> Reply-To: "users@kafka.apache.org<ma...@kafka.apache.org>>" <
> users@kafka.apache.org<ma...@kafka.apache.org>>>
> Date: Monday, June 8, 2015 at 12:20 PM
> To: "users@kafka.apache.org<ma...@kafka.apache.org>>" <
> users@kafka.apache.org<ma...@kafka.apache.org>>>
> Subject: Re: High CPU usage for idle kafka server
>
> Seems to be LogCleaner and FetchRequest threads.
> Attached is the profiling screenshot
>
> On Fri, Jun 5, 2015 at 3:06 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
> <ma...@linkedin.com.invalid>>> wrote:
> Has this to do with KAFKA-1461?
> Can you see which thread is taking a lot of cpu? Some jconsole plugin can
> get that information.
>
> Jiangjie (Becket) Qin
>
> On 6/5/15, 2:57 PM, "pundlik.anuja" <pu...@gmail.com><mailto:
> pundlik.anuja@gmail.com<ma...@gmail.com>>> wrote:
>
> >Hi Jay,
> >
> >Good to hear from you. I met you at the kafka meetup at linkedin.
> >
> >- No, I am running kafka_2.11-0.8.2.1
> >
> >
> >Are there any logs/ any info that I can provide that will help you
> >understand what could be the issue?
> >
> >Thanks,
> >Anuja
> >
> >On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps <ja...@gmail.com><mailto:
> jay.kreps@gmail.com<ma...@gmail.com>>> wrote:
> >
> >> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are
> >>running
> >> that pre-release version is there?
> >>
> >> -Jay
> >>
> >> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) <
> >> apundlik@cisco.com<ma...@cisco.com>>
> >> > wrote:
> >>
> >> > Hi,
> >> >
> >> > I am using Kafka 0.8.2.1.
> >> > We have 1 zookeeper, 3 kafka brokers.
> >> > We have 9 topics, out of which 1 topic has 18 partitions, while
> >>another
> >> > has 12 partitions. All other topics have 1 partition each.
> >> >
> >> > We see that idle kafka brokers (not carrying any message) are using
> >>more
> >> > than 50% of CPU. See top output below.
> >> >
> >> > Is this a known issue?
> >> >
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
> >> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
> >> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
> >> > 0.4%st
> >> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k
> >>buffers
> >> > Swap: 0k total, 0k used, 0k free, 13382460k
> >>cached
> >> >
> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
> >> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
> >> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
> >> >
> >>
>
>
>
Re: High CPU usage for idle kafka server
Posted by "pundlik.anuja" <pu...@gmail.com>.
Hi Jiangjie,
Resending profiling screenshot for High CPU usage on Idle kafka broker.
Appreciate your help.
Thanks,
Anuja
On Mon, Jun 8, 2015 at 1:45 PM, pundlik.anuja <pu...@gmail.com>
wrote:
> Yes, partly it seems to be due to log cleaner thread.
>
> I changed log cleaning settings for each broker (server*.properties) file.
>
>
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=1
>
> # A size-based retention policy for logs. Segments are pruned from the log
> as long as the remaining
> # segments don't drop below log.retention.bytes.
> #log.retention.bytes=1073741824
>
> # The maximum size of a log segment file. When this size is reached a new
> log segment will be created.
> log.segment.bytes=536870912
>
> # The interval at which log segments are checked to see if they can be
> deleted according
> # to the retention policies
> log.retention.check.interval.ms=10000 (****** This was 1 ms before)
>
> # By default the log cleaner is disabled and the log retention policy will
> default to just delete segments after their retention expires.
> # If log.cleaner.enable=true is set the cleaner will be enabled and
> individual logs can then be marked for log compaction.
> log.cleaner.enable=false (***** this was true before)
>
>
>
> *Question:*
> Per my understanding, setting log.cleaner.enable=false will cause deletion
> of logs instead of compaction. Is that correct? Or will it cause the logs
> to keep on accumulating and filling up the disk space?
>
>
> THanks,
> Anuja
>
> On Mon, Jun 8, 2015 at 1:24 PM, Jay Kreps <ja...@gmail.com> wrote:
>
>> Could it also be that the log cleaner is running? This will definitely use
>> some CPU while the cleaning is occurring (it would attempt to use one cpu
>> per log cleaner thread you configure).
>>
>> -Jay
>>
>> On Mon, Jun 8, 2015 at 1:07 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
>> wrote:
>>
>> > It seems the attachments are lost. But high CPU for ReplicaRetcherThread
>> > might be related to KAFKA-1461. Can you try to apply that patch and see
>> if
>> > it solves problem?
>> >
>> > Thanks,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> > From: "pundlik.anuja" <pundlik.anuja@gmail.com<mailto:
>> > pundlik.anuja@gmail.com>>
>> > Reply-To: "users@kafka.apache.org<ma...@kafka.apache.org>" <
>> > users@kafka.apache.org<ma...@kafka.apache.org>>
>> > Date: Monday, June 8, 2015 at 12:20 PM
>> > To: "users@kafka.apache.org<ma...@kafka.apache.org>" <
>> > users@kafka.apache.org<ma...@kafka.apache.org>>
>> > Subject: Re: High CPU usage for idle kafka server
>> >
>> > Seems to be LogCleaner and FetchRequest threads.
>> > Attached is the profiling screenshot
>> >
>> > On Fri, Jun 5, 2015 at 3:06 PM, Jiangjie Qin <jqin@linkedin.com.invalid
>> > <ma...@linkedin.com.invalid>> wrote:
>> > Has this to do with KAFKA-1461?
>> > Can you see which thread is taking a lot of cpu? Some jconsole plugin
>> can
>> > get that information.
>> >
>> > Jiangjie (Becket) Qin
>> >
>> > On 6/5/15, 2:57 PM, "pundlik.anuja" <pundlik.anuja@gmail.com<mailto:
>> > pundlik.anuja@gmail.com>> wrote:
>> >
>> > >Hi Jay,
>> > >
>> > >Good to hear from you. I met you at the kafka meetup at linkedin.
>> > >
>> > >- No, I am running kafka_2.11-0.8.2.1
>> > >
>> > >
>> > >Are there any logs/ any info that I can provide that will help you
>> > >understand what could be the issue?
>> > >
>> > >Thanks,
>> > >Anuja
>> > >
>> > >On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps <jay.kreps@gmail.com<mailto:
>> > jay.kreps@gmail.com>> wrote:
>> > >
>> > >> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are
>> > >>running
>> > >> that pre-release version is there?
>> > >>
>> > >> -Jay
>> > >>
>> > >> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) <
>> > >> apundlik@cisco.com<ma...@cisco.com>
>> > >> > wrote:
>> > >>
>> > >> > Hi,
>> > >> >
>> > >> > I am using Kafka 0.8.2.1.
>> > >> > We have 1 zookeeper, 3 kafka brokers.
>> > >> > We have 9 topics, out of which 1 topic has 18 partitions, while
>> > >>another
>> > >> > has 12 partitions. All other topics have 1 partition each.
>> > >> >
>> > >> > We see that idle kafka brokers (not carrying any message) are using
>> > >>more
>> > >> > than 50% of CPU. See top output below.
>> > >> >
>> > >> > Is this a known issue?
>> > >> >
>> > >> >
>> > >> > Thanks
>> > >> >
>> > >> >
>> > >> >
>> > >> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
>> > >> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0
>> zombie
>> > >> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi,
>> 0.1%si,
>> > >> > 0.4%st
>> > >> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k
>> > >>buffers
>> > >> > Swap: 0k total, 0k used, 0k free, 13382460k
>> > >>cached
>> > >> >
>> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> > >> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
>> > >> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
>> > >> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
>> > >> >
>> > >>
>> >
>> >
>> >
>>
>
>
Re: High CPU usage for idle kafka server
Posted by "pundlik.anuja" <pu...@gmail.com>.
Yes, partly it seems to be due to log cleaner thread.
I changed log cleaning settings for each broker (server*.properties) file.
# The minimum age of a log file to be eligible for deletion
log.retention.hours=1
# A size-based retention policy for logs. Segments are pruned from the log
as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824
# The maximum size of a log segment file. When this size is reached a new
log segment will be created.
log.segment.bytes=536870912
# The interval at which log segments are checked to see if they can be
deleted according
# to the retention policies
log.retention.check.interval.ms=10000 (****** This was 1 ms before)
# By default the log cleaner is disabled and the log retention policy will
default to just delete segments after their retention expires.
# If log.cleaner.enable=true is set the cleaner will be enabled and
individual logs can then be marked for log compaction.
log.cleaner.enable=false (***** this was true before)
*Question:*
Per my understanding, setting log.cleaner.enable=false will cause deletion
of logs instead of compaction. Is that correct? Or will it cause the logs
to keep on accumulating and filling up the disk space?
THanks,
Anuja
On Mon, Jun 8, 2015 at 1:24 PM, Jay Kreps <ja...@gmail.com> wrote:
> Could it also be that the log cleaner is running? This will definitely use
> some CPU while the cleaning is occurring (it would attempt to use one cpu
> per log cleaner thread you configure).
>
> -Jay
>
> On Mon, Jun 8, 2015 at 1:07 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
> wrote:
>
> > It seems the attachments are lost. But high CPU for ReplicaRetcherThread
> > might be related to KAFKA-1461. Can you try to apply that patch and see
> if
> > it solves problem?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > From: "pundlik.anuja" <pundlik.anuja@gmail.com<mailto:
> > pundlik.anuja@gmail.com>>
> > Reply-To: "users@kafka.apache.org<ma...@kafka.apache.org>" <
> > users@kafka.apache.org<ma...@kafka.apache.org>>
> > Date: Monday, June 8, 2015 at 12:20 PM
> > To: "users@kafka.apache.org<ma...@kafka.apache.org>" <
> > users@kafka.apache.org<ma...@kafka.apache.org>>
> > Subject: Re: High CPU usage for idle kafka server
> >
> > Seems to be LogCleaner and FetchRequest threads.
> > Attached is the profiling screenshot
> >
> > On Fri, Jun 5, 2015 at 3:06 PM, Jiangjie Qin <jqin@linkedin.com.invalid
> > <ma...@linkedin.com.invalid>> wrote:
> > Has this to do with KAFKA-1461?
> > Can you see which thread is taking a lot of cpu? Some jconsole plugin can
> > get that information.
> >
> > Jiangjie (Becket) Qin
> >
> > On 6/5/15, 2:57 PM, "pundlik.anuja" <pundlik.anuja@gmail.com<mailto:
> > pundlik.anuja@gmail.com>> wrote:
> >
> > >Hi Jay,
> > >
> > >Good to hear from you. I met you at the kafka meetup at linkedin.
> > >
> > >- No, I am running kafka_2.11-0.8.2.1
> > >
> > >
> > >Are there any logs/ any info that I can provide that will help you
> > >understand what could be the issue?
> > >
> > >Thanks,
> > >Anuja
> > >
> > >On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps <jay.kreps@gmail.com<mailto:
> > jay.kreps@gmail.com>> wrote:
> > >
> > >> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are
> > >>running
> > >> that pre-release version is there?
> > >>
> > >> -Jay
> > >>
> > >> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) <
> > >> apundlik@cisco.com<ma...@cisco.com>
> > >> > wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I am using Kafka 0.8.2.1.
> > >> > We have 1 zookeeper, 3 kafka brokers.
> > >> > We have 9 topics, out of which 1 topic has 18 partitions, while
> > >>another
> > >> > has 12 partitions. All other topics have 1 partition each.
> > >> >
> > >> > We see that idle kafka brokers (not carrying any message) are using
> > >>more
> > >> > than 50% of CPU. See top output below.
> > >> >
> > >> > Is this a known issue?
> > >> >
> > >> >
> > >> > Thanks
> > >> >
> > >> >
> > >> >
> > >> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
> > >> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
> > >> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi,
> 0.1%si,
> > >> > 0.4%st
> > >> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k
> > >>buffers
> > >> > Swap: 0k total, 0k used, 0k free, 13382460k
> > >>cached
> > >> >
> > >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > >> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
> > >> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
> > >> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
> > >> >
> > >>
> >
> >
> >
>
Re: High CPU usage for idle kafka server
Posted by Jay Kreps <ja...@gmail.com>.
Could it also be that the log cleaner is running? This will definitely use
some CPU while the cleaning is occurring (it would attempt to use one cpu
per log cleaner thread you configure).
-Jay
On Mon, Jun 8, 2015 at 1:07 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> It seems the attachments are lost. But high CPU for ReplicaRetcherThread
> might be related to KAFKA-1461. Can you try to apply that patch and see if
> it solves problem?
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> From: "pundlik.anuja" <pundlik.anuja@gmail.com<mailto:
> pundlik.anuja@gmail.com>>
> Reply-To: "users@kafka.apache.org<ma...@kafka.apache.org>" <
> users@kafka.apache.org<ma...@kafka.apache.org>>
> Date: Monday, June 8, 2015 at 12:20 PM
> To: "users@kafka.apache.org<ma...@kafka.apache.org>" <
> users@kafka.apache.org<ma...@kafka.apache.org>>
> Subject: Re: High CPU usage for idle kafka server
>
> Seems to be LogCleaner and FetchRequest threads.
> Attached is the profiling screenshot
>
> On Fri, Jun 5, 2015 at 3:06 PM, Jiangjie Qin <jqin@linkedin.com.invalid
> <ma...@linkedin.com.invalid>> wrote:
> Has this to do with KAFKA-1461?
> Can you see which thread is taking a lot of cpu? Some jconsole plugin can
> get that information.
>
> Jiangjie (Becket) Qin
>
> On 6/5/15, 2:57 PM, "pundlik.anuja" <pundlik.anuja@gmail.com<mailto:
> pundlik.anuja@gmail.com>> wrote:
>
> >Hi Jay,
> >
> >Good to hear from you. I met you at the kafka meetup at linkedin.
> >
> >- No, I am running kafka_2.11-0.8.2.1
> >
> >
> >Are there any logs/ any info that I can provide that will help you
> >understand what could be the issue?
> >
> >Thanks,
> >Anuja
> >
> >On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps <jay.kreps@gmail.com<mailto:
> jay.kreps@gmail.com>> wrote:
> >
> >> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are
> >>running
> >> that pre-release version is there?
> >>
> >> -Jay
> >>
> >> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) <
> >> apundlik@cisco.com<ma...@cisco.com>
> >> > wrote:
> >>
> >> > Hi,
> >> >
> >> > I am using Kafka 0.8.2.1.
> >> > We have 1 zookeeper, 3 kafka brokers.
> >> > We have 9 topics, out of which 1 topic has 18 partitions, while
> >>another
> >> > has 12 partitions. All other topics have 1 partition each.
> >> >
> >> > We see that idle kafka brokers (not carrying any message) are using
> >>more
> >> > than 50% of CPU. See top output below.
> >> >
> >> > Is this a known issue?
> >> >
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
> >> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
> >> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
> >> > 0.4%st
> >> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k
> >>buffers
> >> > Swap: 0k total, 0k used, 0k free, 13382460k
> >>cached
> >> >
> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
> >> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
> >> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
> >> >
> >>
>
>
>
Re: High CPU usage for idle kafka server
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
It seems the attachments are lost. But high CPU for ReplicaRetcherThread might be related to KAFKA-1461. Can you try to apply that patch and see if it solves problem?
Thanks,
Jiangjie (Becket) Qin
From: "pundlik.anuja" <pu...@gmail.com>>
Reply-To: "users@kafka.apache.org<ma...@kafka.apache.org>" <us...@kafka.apache.org>>
Date: Monday, June 8, 2015 at 12:20 PM
To: "users@kafka.apache.org<ma...@kafka.apache.org>" <us...@kafka.apache.org>>
Subject: Re: High CPU usage for idle kafka server
Seems to be LogCleaner and FetchRequest threads.
Attached is the profiling screenshot
On Fri, Jun 5, 2015 at 3:06 PM, Jiangjie Qin <jq...@linkedin.com.invalid>> wrote:
Has this to do with KAFKA-1461?
Can you see which thread is taking a lot of cpu? Some jconsole plugin can
get that information.
Jiangjie (Becket) Qin
On 6/5/15, 2:57 PM, "pundlik.anuja" <pu...@gmail.com>> wrote:
>Hi Jay,
>
>Good to hear from you. I met you at the kafka meetup at linkedin.
>
>- No, I am running kafka_2.11-0.8.2.1
>
>
>Are there any logs/ any info that I can provide that will help you
>understand what could be the issue?
>
>Thanks,
>Anuja
>
>On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps <ja...@gmail.com>> wrote:
>
>> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are
>>running
>> that pre-release version is there?
>>
>> -Jay
>>
>> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) <
>> apundlik@cisco.com<ma...@cisco.com>
>> > wrote:
>>
>> > Hi,
>> >
>> > I am using Kafka 0.8.2.1.
>> > We have 1 zookeeper, 3 kafka brokers.
>> > We have 9 topics, out of which 1 topic has 18 partitions, while
>>another
>> > has 12 partitions. All other topics have 1 partition each.
>> >
>> > We see that idle kafka brokers (not carrying any message) are using
>>more
>> > than 50% of CPU. See top output below.
>> >
>> > Is this a known issue?
>> >
>> >
>> > Thanks
>> >
>> >
>> >
>> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
>> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
>> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
>> > 0.4%st
>> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k
>>buffers
>> > Swap: 0k total, 0k used, 0k free, 13382460k
>>cached
>> >
>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
>> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
>> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
>> >
>>
Re: High CPU usage for idle kafka server
Posted by "pundlik.anuja" <pu...@gmail.com>.
Seems to be LogCleaner and FetchRequest threads.
Attached is the profiling screenshot
On Fri, Jun 5, 2015 at 3:06 PM, Jiangjie Qin <jq...@linkedin.com.invalid>
wrote:
> Has this to do with KAFKA-1461?
> Can you see which thread is taking a lot of cpu? Some jconsole plugin can
> get that information.
>
> Jiangjie (Becket) Qin
>
> On 6/5/15, 2:57 PM, "pundlik.anuja" <pu...@gmail.com> wrote:
>
> >Hi Jay,
> >
> >Good to hear from you. I met you at the kafka meetup at linkedin.
> >
> >- No, I am running kafka_2.11-0.8.2.1
> >
> >
> >Are there any logs/ any info that I can provide that will help you
> >understand what could be the issue?
> >
> >Thanks,
> >Anuja
> >
> >On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps <ja...@gmail.com> wrote:
> >
> >> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are
> >>running
> >> that pre-release version is there?
> >>
> >> -Jay
> >>
> >> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) <
> >> apundlik@cisco.com
> >> > wrote:
> >>
> >> > Hi,
> >> >
> >> > I am using Kafka 0.8.2.1.
> >> > We have 1 zookeeper, 3 kafka brokers.
> >> > We have 9 topics, out of which 1 topic has 18 partitions, while
> >>another
> >> > has 12 partitions. All other topics have 1 partition each.
> >> >
> >> > We see that idle kafka brokers (not carrying any message) are using
> >>more
> >> > than 50% of CPU. See top output below.
> >> >
> >> > Is this a known issue?
> >> >
> >> >
> >> > Thanks
> >> >
> >> >
> >> >
> >> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
> >> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
> >> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
> >> > 0.4%st
> >> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k
> >>buffers
> >> > Swap: 0k total, 0k used, 0k free, 13382460k
> >>cached
> >> >
> >> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> >> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
> >> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
> >> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
> >> >
> >>
>
>
Re: High CPU usage for idle kafka server
Posted by Jiangjie Qin <jq...@linkedin.com.INVALID>.
Has this to do with KAFKA-1461?
Can you see which thread is taking a lot of cpu? Some jconsole plugin can
get that information.
Jiangjie (Becket) Qin
On 6/5/15, 2:57 PM, "pundlik.anuja" <pu...@gmail.com> wrote:
>Hi Jay,
>
>Good to hear from you. I met you at the kafka meetup at linkedin.
>
>- No, I am running kafka_2.11-0.8.2.1
>
>
>Are there any logs/ any info that I can provide that will help you
>understand what could be the issue?
>
>Thanks,
>Anuja
>
>On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps <ja...@gmail.com> wrote:
>
>> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are
>>running
>> that pre-release version is there?
>>
>> -Jay
>>
>> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) <
>> apundlik@cisco.com
>> > wrote:
>>
>> > Hi,
>> >
>> > I am using Kafka 0.8.2.1.
>> > We have 1 zookeeper, 3 kafka brokers.
>> > We have 9 topics, out of which 1 topic has 18 partitions, while
>>another
>> > has 12 partitions. All other topics have 1 partition each.
>> >
>> > We see that idle kafka brokers (not carrying any message) are using
>>more
>> > than 50% of CPU. See top output below.
>> >
>> > Is this a known issue?
>> >
>> >
>> > Thanks
>> >
>> >
>> >
>> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
>> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
>> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
>> > 0.4%st
>> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k
>>buffers
>> > Swap: 0k total, 0k used, 0k free, 13382460k
>>cached
>> >
>> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
>> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
>> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
>> >
>>
Re: High CPU usage for idle kafka server
Posted by "pundlik.anuja" <pu...@gmail.com>.
Hi Jay,
Good to hear from you. I met you at the kafka meetup at linkedin.
- No, I am running kafka_2.11-0.8.2.1
Are there any logs/ any info that I can provide that will help you
understand what could be the issue?
Thanks,
Anuja
On Fri, Jun 5, 2015 at 2:36 PM, Jay Kreps <ja...@gmail.com> wrote:
> This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are running
> that pre-release version is there?
>
> -Jay
>
> On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) <
> apundlik@cisco.com
> > wrote:
>
> > Hi,
> >
> > I am using Kafka 0.8.2.1.
> > We have 1 zookeeper, 3 kafka brokers.
> > We have 9 topics, out of which 1 topic has 18 partitions, while another
> > has 12 partitions. All other topics have 1 partition each.
> >
> > We see that idle kafka brokers (not carrying any message) are using more
> > than 50% of CPU. See top output below.
> >
> > Is this a known issue?
> >
> >
> > Thanks
> >
> >
> >
> > top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
> > Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
> > Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
> > 0.4%st
> > Mem: 65974296k total, 22310524k used, 43663772k free, 112688k buffers
> > Swap: 0k total, 0k used, 0k free, 13382460k cached
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
> > 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
> > 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
> >
>
Re: High CPU usage for idle kafka server
Posted by Jay Kreps <ja...@gmail.com>.
This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are running
that pre-release version is there?
-Jay
On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) <apundlik@cisco.com
> wrote:
> Hi,
>
> I am using Kafka 0.8.2.1.
> We have 1 zookeeper, 3 kafka brokers.
> We have 9 topics, out of which 1 topic has 18 partitions, while another
> has 12 partitions. All other topics have 1 partition each.
>
> We see that idle kafka brokers (not carrying any message) are using more
> than 50% of CPU. See top output below.
>
> Is this a known issue?
>
>
> Thanks
>
>
>
> top - 04:42:30 up 2:07, 1 user, load average: 1.50, 1.31, 0.92
> Tasks: 177 total, 1 running, 176 sleeping, 0 stopped, 0 zombie
> Cpu(s): 13.5%us, 4.5%sy, 0.0%ni, 81.3%id, 0.2%wa, 0.0%hi, 0.1%si,
> 0.4%st
> Mem: 65974296k total, 22310524k used, 43663772k free, 112688k buffers
> Swap: 0k total, 0k used, 0k free, 13382460k cached
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 9295 wae 20 0 5212m 894m 12m S 62 1.4 22:50.99 java
> 9323 wae 20 0 5502m 894m 12m S 56 1.4 24:28.69 java
> 9353 wae 20 0 5072m 896m 12m S 54 1.4 17:04.31 java
>