You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Dennis Cammer <de...@gmail.com> on 2015/03/12 08:34:59 UTC
Worker process shows higher than expected CPU usage on an idle system
We currently use storm 0.9.2 and upgraded from Redhat Enterprise Linux 4 to
Redhat Enterprise Linux 6. After the upgrade, we saw that the CPU usage
when the system is idle is a lot higher on the new system. It's now around
~55%, when it was around ~14% on RHEL4 with the same topology.
Looking at the threads using the CPU, there is no single thread using a lot
of CPU, but the CPU usage is distributed between many threads. Increasing "
topology.sleep.spout.wait.strategy.time.ms" from 1 to 25 ms helped to bring
down the CPU from ~55% to ~30% with no single thread using more than 1.4%
CPU (most are using 0.3%).
Has anybody seen something similar?
If you are using RHEL6, have you seen higher than expected CPU for an idle
worker? Is this expected?
Thanks,
Dennis
Re: Worker process shows higher than expected CPU usage on an idle system
Posted by Dennis Cammer <de...@gmail.com>.
We have tried the system with load. With RHEL4 we have events being
processed in 126ms (our "topology.trident.batch.emit.interval.millis" is
set to 100ms), on RHEL6 the average is 8 seconds. We are trying different
settings to isolate the problems, and suggestions are welcome.
The troubling part to me is that storm and RHEL6 are both popular platforms
and it's hard to imagine that other people are not seeing the same thing.
We've been looking into our code, but even when we simplify our topology to
a minimum, we still see "highish" CPU usage when idle (25% of one CPU).
Dennis
On Mon, Mar 16, 2015 at 10:28 AM, Dennis Cammer <de...@gmail.com>
wrote:
> I've just tried disabling Transparent Huge Pages (THP), and there was no
> noticeable difference in the CPU usage.
>
> We've been running tests and it seems that RHEL6 is better about
> respecting the time of the sleep statements in the code; when the code
> says, sleep 1ms, RHEL4 tends to be more sloppy and wake up late, while
> RHEL6 will wake up the thread close to the time specified. That explains
> why increasing the "topology.sleep.spout.wait.strategy.time.ms" from 1 to
> 25 ms cut the CPU usage in almost half.
> The comparison is being done using a machine running in VMWare; not sure
> that matters.
>
> Thanks,
> Dennis
>
> On Sun, Mar 15, 2015 at 9:24 PM, Dennis Cammer <de...@gmail.com>
> wrote:
>
>> As for the number of spout threads, we are usually using 5 spouts + DRPC.
>> This is a 4 core machine, but using 30-50% of one CPU while idle is
>> suspicious that something is wrong.
>>
>> Thanks, I haven't checked or disabled THP. I'll give that a try.
>>
>> On Sun, Mar 15, 2015 at 9:05 PM, 임정택 <ka...@gmail.com> wrote:
>>
>>> Did you disable THP? If you didn't, it would be better to give it a try.
>>> (Sorry, but I don't find a relationship between these, but THP shows CPU
>>> issues from many places.)
>>> https://twitter.com/mattsta/status/529327782351097856
>>>
>>> 2015-03-16 8:03 GMT+09:00 Dennis Cammer <de...@gmail.com>:
>>>
>>>> I haven't heard anything back on this. Does anybody else see the worker
>>>> process using 25-50% of one CPU when the system is completely idle when
>>>> using RHEL6 or CentOS 6? Is that expected? I'd also appreciate a "no, I
>>>> don't see that".
>>>>
>>>> Thanks,
>>>> Dennis
>>>>
>>>> On Thu, Mar 12, 2015 at 3:34 AM, Dennis Cammer <de...@gmail.com>
>>>> wrote:
>>>>
>>>>> We currently use storm 0.9.2 and upgraded from Redhat Enterprise Linux
>>>>> 4 to Redhat Enterprise Linux 6. After the upgrade, we saw that the CPU
>>>>> usage when the system is idle is a lot higher on the new system. It's now
>>>>> around ~55%, when it was around ~14% on RHEL4 with the same topology.
>>>>> Looking at the threads using the CPU, there is no single thread using
>>>>> a lot of CPU, but the CPU usage is distributed between many threads.
>>>>> Increasing "topology.sleep.spout.wait.strategy.time.ms" from 1 to 25
>>>>> ms helped to bring down the CPU from ~55% to ~30% with no single thread
>>>>> using more than 1.4% CPU (most are using 0.3%).
>>>>>
>>>>> Has anybody seen something similar?
>>>>> If you are using RHEL6, have you seen higher than expected CPU for an
>>>>> idle worker? Is this expected?
>>>>>
>>>>> Thanks,
>>>>> Dennis
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Name : 임 정택
>>> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
>>> Twitter : http://twitter.com/heartsavior
>>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>>
>>
>>
>
Re: Worker process shows higher than expected CPU usage on an idle system
Posted by Dennis Cammer <de...@gmail.com>.
I've just tried disabling Transparent Huge Pages (THP), and there was no
noticeable difference in the CPU usage.
We've been running tests and it seems that RHEL6 is better about respecting
the time of the sleep statements in the code; when the code says, sleep
1ms, RHEL4 tends to be more sloppy and wake up late, while RHEL6 will wake
up the thread close to the time specified. That explains why increasing the
"topology.sleep.spout.wait.strategy.time.ms" from 1 to 25 ms cut the CPU
usage in almost half.
The comparison is being done using a machine running in VMWare; not sure
that matters.
Thanks,
Dennis
On Sun, Mar 15, 2015 at 9:24 PM, Dennis Cammer <de...@gmail.com>
wrote:
> As for the number of spout threads, we are usually using 5 spouts + DRPC.
> This is a 4 core machine, but using 30-50% of one CPU while idle is
> suspicious that something is wrong.
>
> Thanks, I haven't checked or disabled THP. I'll give that a try.
>
> On Sun, Mar 15, 2015 at 9:05 PM, 임정택 <ka...@gmail.com> wrote:
>
>> Did you disable THP? If you didn't, it would be better to give it a try.
>> (Sorry, but I don't find a relationship between these, but THP shows CPU
>> issues from many places.)
>> https://twitter.com/mattsta/status/529327782351097856
>>
>> 2015-03-16 8:03 GMT+09:00 Dennis Cammer <de...@gmail.com>:
>>
>>> I haven't heard anything back on this. Does anybody else see the worker
>>> process using 25-50% of one CPU when the system is completely idle when
>>> using RHEL6 or CentOS 6? Is that expected? I'd also appreciate a "no, I
>>> don't see that".
>>>
>>> Thanks,
>>> Dennis
>>>
>>> On Thu, Mar 12, 2015 at 3:34 AM, Dennis Cammer <de...@gmail.com>
>>> wrote:
>>>
>>>> We currently use storm 0.9.2 and upgraded from Redhat Enterprise Linux
>>>> 4 to Redhat Enterprise Linux 6. After the upgrade, we saw that the CPU
>>>> usage when the system is idle is a lot higher on the new system. It's now
>>>> around ~55%, when it was around ~14% on RHEL4 with the same topology.
>>>> Looking at the threads using the CPU, there is no single thread using a
>>>> lot of CPU, but the CPU usage is distributed between many threads.
>>>> Increasing "topology.sleep.spout.wait.strategy.time.ms" from 1 to 25
>>>> ms helped to bring down the CPU from ~55% to ~30% with no single thread
>>>> using more than 1.4% CPU (most are using 0.3%).
>>>>
>>>> Has anybody seen something similar?
>>>> If you are using RHEL6, have you seen higher than expected CPU for an
>>>> idle worker? Is this expected?
>>>>
>>>> Thanks,
>>>> Dennis
>>>>
>>>
>>>
>>
>>
>> --
>> Name : 임 정택
>> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
>> Twitter : http://twitter.com/heartsavior
>> LinkedIn : http://www.linkedin.com/in/heartsavior
>>
>
>
Re: Worker process shows higher than expected CPU usage on an idle system
Posted by Dennis Cammer <de...@gmail.com>.
As for the number of spout threads, we are usually using 5 spouts + DRPC.
This is a 4 core machine, but using 30-50% of one CPU while idle is
suspicious that something is wrong.
Thanks, I haven't checked or disabled THP. I'll give that a try.
On Sun, Mar 15, 2015 at 9:05 PM, 임정택 <ka...@gmail.com> wrote:
> Did you disable THP? If you didn't, it would be better to give it a try.
> (Sorry, but I don't find a relationship between these, but THP shows CPU
> issues from many places.)
> https://twitter.com/mattsta/status/529327782351097856
>
> 2015-03-16 8:03 GMT+09:00 Dennis Cammer <de...@gmail.com>:
>
>> I haven't heard anything back on this. Does anybody else see the worker
>> process using 25-50% of one CPU when the system is completely idle when
>> using RHEL6 or CentOS 6? Is that expected? I'd also appreciate a "no, I
>> don't see that".
>>
>> Thanks,
>> Dennis
>>
>> On Thu, Mar 12, 2015 at 3:34 AM, Dennis Cammer <de...@gmail.com>
>> wrote:
>>
>>> We currently use storm 0.9.2 and upgraded from Redhat Enterprise Linux 4
>>> to Redhat Enterprise Linux 6. After the upgrade, we saw that the CPU usage
>>> when the system is idle is a lot higher on the new system. It's now around
>>> ~55%, when it was around ~14% on RHEL4 with the same topology.
>>> Looking at the threads using the CPU, there is no single thread using a
>>> lot of CPU, but the CPU usage is distributed between many threads.
>>> Increasing "topology.sleep.spout.wait.strategy.time.ms" from 1 to 25 ms
>>> helped to bring down the CPU from ~55% to ~30% with no single thread using
>>> more than 1.4% CPU (most are using 0.3%).
>>>
>>> Has anybody seen something similar?
>>> If you are using RHEL6, have you seen higher than expected CPU for an
>>> idle worker? Is this expected?
>>>
>>> Thanks,
>>> Dennis
>>>
>>
>>
>
>
> --
> Name : 임 정택
> Blog : http://www.heartsavior.net / http://dev.heartsavior.net
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>
Re: Worker process shows higher than expected CPU usage on an idle system
Posted by 임정택 <ka...@gmail.com>.
Did you disable THP? If you didn't, it would be better to give it a try.
(Sorry, but I don't find a relationship between these, but THP shows CPU
issues from many places.)
https://twitter.com/mattsta/status/529327782351097856
2015-03-16 8:03 GMT+09:00 Dennis Cammer <de...@gmail.com>:
> I haven't heard anything back on this. Does anybody else see the worker
> process using 25-50% of one CPU when the system is completely idle when
> using RHEL6 or CentOS 6? Is that expected? I'd also appreciate a "no, I
> don't see that".
>
> Thanks,
> Dennis
>
> On Thu, Mar 12, 2015 at 3:34 AM, Dennis Cammer <de...@gmail.com>
> wrote:
>
>> We currently use storm 0.9.2 and upgraded from Redhat Enterprise Linux 4
>> to Redhat Enterprise Linux 6. After the upgrade, we saw that the CPU usage
>> when the system is idle is a lot higher on the new system. It's now around
>> ~55%, when it was around ~14% on RHEL4 with the same topology.
>> Looking at the threads using the CPU, there is no single thread using a
>> lot of CPU, but the CPU usage is distributed between many threads.
>> Increasing "topology.sleep.spout.wait.strategy.time.ms" from 1 to 25 ms
>> helped to bring down the CPU from ~55% to ~30% with no single thread using
>> more than 1.4% CPU (most are using 0.3%).
>>
>> Has anybody seen something similar?
>> If you are using RHEL6, have you seen higher than expected CPU for an
>> idle worker? Is this expected?
>>
>> Thanks,
>> Dennis
>>
>
>
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
Re: Worker process shows higher than expected CPU usage on an idle system
Posted by Nathan Leung <nc...@gmail.com>.
I don't (RHEL 6). How many spout threads do you have? I don't have nearly
that many so maybe that's why. Also how many cores do your supervisor
nodes have?
On Sun, Mar 15, 2015 at 7:03 PM, Dennis Cammer <de...@gmail.com>
wrote:
> I haven't heard anything back on this. Does anybody else see the worker
> process using 25-50% of one CPU when the system is completely idle when
> using RHEL6 or CentOS 6? Is that expected? I'd also appreciate a "no, I
> don't see that".
>
> Thanks,
> Dennis
>
> On Thu, Mar 12, 2015 at 3:34 AM, Dennis Cammer <de...@gmail.com>
> wrote:
>
>> We currently use storm 0.9.2 and upgraded from Redhat Enterprise Linux 4
>> to Redhat Enterprise Linux 6. After the upgrade, we saw that the CPU usage
>> when the system is idle is a lot higher on the new system. It's now around
>> ~55%, when it was around ~14% on RHEL4 with the same topology.
>> Looking at the threads using the CPU, there is no single thread using a
>> lot of CPU, but the CPU usage is distributed between many threads.
>> Increasing "topology.sleep.spout.wait.strategy.time.ms" from 1 to 25 ms
>> helped to bring down the CPU from ~55% to ~30% with no single thread using
>> more than 1.4% CPU (most are using 0.3%).
>>
>> Has anybody seen something similar?
>> If you are using RHEL6, have you seen higher than expected CPU for an
>> idle worker? Is this expected?
>>
>> Thanks,
>> Dennis
>>
>
>
Re: Worker process shows higher than expected CPU usage on an idle system
Posted by Dennis Cammer <de...@gmail.com>.
I haven't heard anything back on this. Does anybody else see the worker
process using 25-50% of one CPU when the system is completely idle when
using RHEL6 or CentOS 6? Is that expected? I'd also appreciate a "no, I
don't see that".
Thanks,
Dennis
On Thu, Mar 12, 2015 at 3:34 AM, Dennis Cammer <de...@gmail.com>
wrote:
> We currently use storm 0.9.2 and upgraded from Redhat Enterprise Linux 4
> to Redhat Enterprise Linux 6. After the upgrade, we saw that the CPU usage
> when the system is idle is a lot higher on the new system. It's now around
> ~55%, when it was around ~14% on RHEL4 with the same topology.
> Looking at the threads using the CPU, there is no single thread using a
> lot of CPU, but the CPU usage is distributed between many threads.
> Increasing "topology.sleep.spout.wait.strategy.time.ms" from 1 to 25 ms
> helped to bring down the CPU from ~55% to ~30% with no single thread using
> more than 1.4% CPU (most are using 0.3%).
>
> Has anybody seen something similar?
> If you are using RHEL6, have you seen higher than expected CPU for an idle
> worker? Is this expected?
>
> Thanks,
> Dennis
>