You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Igor Galić <i....@brainsware.org> on 2011/10/02 20:21:41 UTC
Re: High idle cpu utilization on RHEL6
----- Original Message -----
> On Wed, Sep 28, 2011 at 3:35 PM, Brian Geffon <br...@gmail.com>
> wrote:
>
> > Hi,
> > I'm encountering something strange with ATS3.0.1 on Red Hat
> > Enterprise
> > Linux 6, using a vanilla build with no modules enabled and the
> > default
> > records.config /w zero entries in the remap file, ATS is idling at
> > very high
> > CPU (around 15-20%).
> >
> > root 13509 0.0 0.0 58512 2392 ? Ss 19:15 0:00
> > /usr/local/ats3.0.1plain//bin/traffic_cop
> > nobody 13511 0.1 0.0 480072 16632 ? Sl 19:15 0:00
> > /usr/local/ats3.0.1plain/bin/traffic_manager
> > nobody 13521 ***16.9*** 0.1 1584628 114224 ? Sl 19:15
> > 2:06
> > /usr/local/ats3.0.1plain/bin/traffic_server -M -A,7:X
> >
> > So I used strace to try to determine what might be causing this,
> > and here
> > is what i've found:
> >
> > [root@machine]# strace -c -p 13521
> > Process 13521 attached - interrupt to quit
> > ^CProcess 13521 detached
> > % time seconds usecs/call calls errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00 3.589451 796 4510 epoll_wait
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00 3.589451 4510 total
> > [root@machine]#
> >
> >
> > It appears that it's entirely epoll_wait, and each call is taking
> > 796
> > microseconds! So I have to concerns with this, first, why would
> > epoll_wait
> > take such a long amount of time, 796 microseconds seems like a long
> > time,
> > and more importantly, how could it possibly be called so
> > frequently, does
> > ATS use a short timeout when doing epoll_waits?
> >
> > I would really appreciate any feedback regarding this, has anyone
> > else
> > experienced this? Is there anywhere else I might look to determine
> > the cause
> > of this? Could this be classified as _normal_ behavior?
> >
> >
> >
> It is working as designed. The epoll timeout is set to 0 or 10 msec
> on
> linux. Could there be very little or no load going through this
> instance?
> That would explain why you are only seeing epoll_wait. In that case
> there
> isn't really anything wrong, the process is just burning through
> epoll_waits
> looking for something to do when nothing is available. Once the
> process
> starts taking enough traffic, you will start to see user space and
> other
> kernel functions start to take cpu time.
That still doesn't quite explain the high load on an Idle system.
Brian: Do you happen to have SELinux enabled?
> Sridhar
i
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 571B 8B8A FC97 266D BDA3 EF6F 43AD 80A4 5779 3257
Re: High idle cpu utilization on RHEL6
Posted by Leif Hedstrom <zw...@apache.org>.
On 10/13/2011 02:00 AM, Brian Geffon wrote:
> Hi Igor,
>
> No I am not using SELinux, any other ideas on what might be causing
> this, it seems very strange. I suppose it should be worth noting this
> is an 8 core machine, could the thread auto scaling just be allocating
> more resources to cover the idle threads?
So, what percentage of the CPU is this using? Does it matter? If you
need to recover some of it, yes, you can turn down the number of worker
threads. I see the same behaviour on older kernels as well, but not on
modern ones. On my old kernel, with a couple of ATS threads, on idle it
uses 1.5% of the CPUs, which is completely negligible. Now, if you
expect load on the box, I wouldn't turn down the number of threads,
because if you do, you will not be able to use all 8 CPUs.
As been pointed out already, once the system takes some real load, this
is a non-issue.
-- leif
Re: High idle cpu utilization on RHEL6
Posted by Leif Hedstrom <zw...@apache.org>.
On 10/13/2011 02:00 AM, Brian Geffon wrote:
> Hi Igor,
>
> No I am not using SELinux, any other ideas on what might be causing
> this, it seems very strange. I suppose it should be worth noting this
> is an 8 core machine, could the thread auto scaling just be allocating
> more resources to cover the idle threads?
So, what percentage of the CPU is this using? Does it matter? If you
need to recover some of it, yes, you can turn down the number of worker
threads. I see the same behaviour on older kernels as well, but not on
modern ones. On my old kernel, with a couple of ATS threads, on idle it
uses 1.5% of the CPUs, which is completely negligible. Now, if you
expect load on the box, I wouldn't turn down the number of threads,
because if you do, you will not be able to use all 8 CPUs.
As been pointed out already, once the system takes some real load, this
is a non-issue.
-- leif
Re: High idle cpu utilization on RHEL6
Posted by Brian Geffon <br...@gmail.com>.
Hi Igor,
No I am not using SELinux, any other ideas on what might be causing
this, it seems very strange. I suppose it should be worth noting this
is an 8 core machine, could the thread auto scaling just be allocating
more resources to cover the idle threads?
Thanks!
On Oct 2, 2011, at 11:22 AM, "Igor Galić" <i....@brainsware.org> wrote:
>
>
> ----- Original Message -----
>> On Wed, Sep 28, 2011 at 3:35 PM, Brian Geffon <br...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I'm encountering something strange with ATS3.0.1 on Red Hat
>>> Enterprise
>>> Linux 6, using a vanilla build with no modules enabled and the
>>> default
>>> records.config /w zero entries in the remap file, ATS is idling at
>>> very high
>>> CPU (around 15-20%).
>>>
>>> root 13509 0.0 0.0 58512 2392 ? Ss 19:15 0:00
>>> /usr/local/ats3.0.1plain//bin/traffic_cop
>>> nobody 13511 0.1 0.0 480072 16632 ? Sl 19:15 0:00
>>> /usr/local/ats3.0.1plain/bin/traffic_manager
>>> nobody 13521 ***16.9*** 0.1 1584628 114224 ? Sl 19:15
>>> 2:06
>>> /usr/local/ats3.0.1plain/bin/traffic_server -M -A,7:X
>>>
>>> So I used strace to try to determine what might be causing this,
>>> and here
>>> is what i've found:
>>>
>>> [root@machine]# strace -c -p 13521
>>> Process 13521 attached - interrupt to quit
>>> ^CProcess 13521 detached
>>> % time seconds usecs/call calls errors syscall
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 100.00 3.589451 796 4510 epoll_wait
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 100.00 3.589451 4510 total
>>> [root@machine]#
>>>
>>>
>>> It appears that it's entirely epoll_wait, and each call is taking
>>> 796
>>> microseconds! So I have to concerns with this, first, why would
>>> epoll_wait
>>> take such a long amount of time, 796 microseconds seems like a long
>>> time,
>>> and more importantly, how could it possibly be called so
>>> frequently, does
>>> ATS use a short timeout when doing epoll_waits?
>>>
>>> I would really appreciate any feedback regarding this, has anyone
>>> else
>>> experienced this? Is there anywhere else I might look to determine
>>> the cause
>>> of this? Could this be classified as _normal_ behavior?
>>>
>>>
>>>
>> It is working as designed. The epoll timeout is set to 0 or 10 msec
>> on
>> linux. Could there be very little or no load going through this
>> instance?
>> That would explain why you are only seeing epoll_wait. In that case
>> there
>> isn't really anything wrong, the process is just burning through
>> epoll_waits
>> looking for something to do when nothing is available. Once the
>> process
>> starts taking enough traffic, you will start to see user space and
>> other
>> kernel functions start to take cpu time.
>
> That still doesn't quite explain the high load on an Idle system.
>
> Brian: Do you happen to have SELinux enabled?
>
>> Sridhar
>
> i
>
> --
> Igor Galić
>
> Tel: +43 (0) 664 886 22 883
> Mail: i.galic@brainsware.org
> URL: http://brainsware.org/
> GPG: 571B 8B8A FC97 266D BDA3 EF6F 43AD 80A4 5779 3257
Re: High idle cpu utilization on RHEL6
Posted by Brian Geffon <br...@gmail.com>.
Hi Igor,
No I am not using SELinux, any other ideas on what might be causing
this, it seems very strange. I suppose it should be worth noting this
is an 8 core machine, could the thread auto scaling just be allocating
more resources to cover the idle threads?
Thanks!
On Oct 2, 2011, at 11:22 AM, "Igor Galić" <i....@brainsware.org> wrote:
>
>
> ----- Original Message -----
>> On Wed, Sep 28, 2011 at 3:35 PM, Brian Geffon <br...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> I'm encountering something strange with ATS3.0.1 on Red Hat
>>> Enterprise
>>> Linux 6, using a vanilla build with no modules enabled and the
>>> default
>>> records.config /w zero entries in the remap file, ATS is idling at
>>> very high
>>> CPU (around 15-20%).
>>>
>>> root 13509 0.0 0.0 58512 2392 ? Ss 19:15 0:00
>>> /usr/local/ats3.0.1plain//bin/traffic_cop
>>> nobody 13511 0.1 0.0 480072 16632 ? Sl 19:15 0:00
>>> /usr/local/ats3.0.1plain/bin/traffic_manager
>>> nobody 13521 ***16.9*** 0.1 1584628 114224 ? Sl 19:15
>>> 2:06
>>> /usr/local/ats3.0.1plain/bin/traffic_server -M -A,7:X
>>>
>>> So I used strace to try to determine what might be causing this,
>>> and here
>>> is what i've found:
>>>
>>> [root@machine]# strace -c -p 13521
>>> Process 13521 attached - interrupt to quit
>>> ^CProcess 13521 detached
>>> % time seconds usecs/call calls errors syscall
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 100.00 3.589451 796 4510 epoll_wait
>>> ------ ----------- ----------- --------- --------- ----------------
>>> 100.00 3.589451 4510 total
>>> [root@machine]#
>>>
>>>
>>> It appears that it's entirely epoll_wait, and each call is taking
>>> 796
>>> microseconds! So I have to concerns with this, first, why would
>>> epoll_wait
>>> take such a long amount of time, 796 microseconds seems like a long
>>> time,
>>> and more importantly, how could it possibly be called so
>>> frequently, does
>>> ATS use a short timeout when doing epoll_waits?
>>>
>>> I would really appreciate any feedback regarding this, has anyone
>>> else
>>> experienced this? Is there anywhere else I might look to determine
>>> the cause
>>> of this? Could this be classified as _normal_ behavior?
>>>
>>>
>>>
>> It is working as designed. The epoll timeout is set to 0 or 10 msec
>> on
>> linux. Could there be very little or no load going through this
>> instance?
>> That would explain why you are only seeing epoll_wait. In that case
>> there
>> isn't really anything wrong, the process is just burning through
>> epoll_waits
>> looking for something to do when nothing is available. Once the
>> process
>> starts taking enough traffic, you will start to see user space and
>> other
>> kernel functions start to take cpu time.
>
> That still doesn't quite explain the high load on an Idle system.
>
> Brian: Do you happen to have SELinux enabled?
>
>> Sridhar
>
> i
>
> --
> Igor Galić
>
> Tel: +43 (0) 664 886 22 883
> Mail: i.galic@brainsware.org
> URL: http://brainsware.org/
> GPG: 571B 8B8A FC97 266D BDA3 EF6F 43AD 80A4 5779 3257