You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by "Jason J. W. Williams" <ja...@gmail.com> on 2015/01/07 00:13:13 UTC
CPU Usage Analysis
I'm seeing CPU spikes sporadically and would like to figure out what
activities (caching, network transfer, etc) in ATS are using the most
CPU so I can tune them. Is there a good way to analyze that?
-J
Re: CPU Usage Analysis
Posted by Daniel Biazus <da...@azion.com>.
Hi Adam,
Thank you for your very useful advice, it works really fine.
perf top
2.36% [kernel] [k] update_sd_lb_stats
2.05% traffic_server [.] _ZN10NetHandler12mainNetEventEiP5Event
1.97% libpcre.so.0.0.1 [.] 0x0000000000010a63
1.84% [kernel] [k] __ticket_spin_lock
1.75% [kernel] [k] __copy_user_nocache
1.66% traffic_server [.] _ZN7EThread13process_eventEP5Eventi
1.52% [e1000e] [k] e1000e_check_ltr_demote
1.24% [kernel] [k] timerqueue_add
1.23% traffic_server [.] 0x000000000031ece0
1.20% libc-2.12.so [.] memcpy
1.11% traffic_server [.] _ZN7EThread7executeEv
1.10% [kernel] [k] fget_light
0.94% libpthread-2.12.so [.] pthread_getspecific
0.93% libcrypto.so.1.0.1e [.] 0x00000000000b0616
0.92% traffic_server [.]
_ZN9LogBuffer20resolve_custom_entryEP12LogFieldListPcS2_S2_illjS1_S2_
0.91% libpthread-2.12.so [.] pthread_mutex_trylock
0.88% libtsutil.so.5.1.0 [.] ink_atomiclist_popall
0.85% libc-2.12.so [.] epoll_wait
0.84% [kernel] [k] __schedule
0.82% [kernel] [k] __alloc_skb
0.75% [kernel] [k] tcp_sendmsg
I used the following parameters:
CONFIG proxy.config.net.sock_send_buffer_size_in INT 65536
CONFIG proxy.config.net.sock_recv_buffer_size_in INT 65536
CONFIG proxy.config.net.sock_option_flag_in INT 1
CONFIG proxy.config.net.sock_send_buffer_size_out INT 65536
CONFIG proxy.config.net.sock_recv_buffer_size_out INT 65536
CONFIG proxy.config.net.sock_option_flag_out INT 1
Thanks & regards,
Daniel Biazus
On Wed, Jan 7, 2015 at 1:57 PM, Adam W. Dace <co...@gmail.com>
wrote:
> If TrafficServer does seem to be spending a lot of CPU time simply writing
> data to the network,
> one thing some people have had luck with in the past is simply to enable
> application-level buffering.
>
> See the documentation, the settings for that are:
>
> proxy.config.net.sock_recv_buffer_size_in
> proxy.config.net.sock_send_buffer_size_in
> proxy.config.net.sock_recv_buffer_size_out
> proxy.config.net.sock_send_buffer_size_out
>
> Hope this helps.
>
> Regards,
>
> Adam
>
> On Wed Jan 07 2015 at 6:11:53 AM Daniel Biazus <da...@azion.com>
> wrote:
>
>> Hi Guys,
>>
>> We have been facing a very similar issue. I got this with with perf top
>> tool:
>>
>> 88.18% traffic_server [.] _Z15write_to_net_
>> ioP10NetHandlerP18UnixNetVConnectionP7EThread
>> 0.32% traffic_server [.] _ZN10NetHandler12mainNetEventEi
>> P5Event
>> 0.30% [kernel] [k] update_sd_lb_stats
>> 0.29% [e1000e] [k] e1000e_check_ltr_demote
>> 0.25% [kernel] [k] __ticket_spin_lock
>> 0.24% traffic_server [.] _ZN7EThread13process_eventEP5Eventi
>> 0.21% [kernel] [k] timerqueue_add
>> 0.17% libc-2.12.so [.] epoll_wait
>> 0.17% libpcre.so.0.0.1 [.] 0x00000000000100dd
>> 0.14% [kernel] [k] __schedule
>>
>> Any thoughts ?
>>
>> Best Regards,
>> Daniel Biazus
>>
>>
>> On Tue, Jan 6, 2015 at 10:36 PM, Jason J. W. Williams <
>> jasonjwwilliams@gmail.com> wrote:
>>
>>> Thanks Sudheer. I was thinking maybe there was some built-in
>>> instrumentation that would help. During the spikes all perf shows me
>>> as the top calls are:
>>>
>>> 9.79% [kernel] [k] _raw_spin_lock
>>> 9.44% [kernel] [k] find_busiest_group
>>>
>>> Which isn't very helpful.
>>>
>>> -J
>>>
>>
>>
>>
>> --
>>
>> Daniel Biazus
>> infrastructure Engineering
>> Azion Technologies
>> Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
>> Miami, USA +1 305 704 8816
>>
>> Quaisquer informações contidas neste e-mail e anexos podem ser
>> confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
>> de utilização deste documento depende de autorização do emissor, sujeito as
>> penalidades cabíveis.
>>
>> Any information in this e-mail and attachments may be confidential and
>> privileged, protected by legal confidentiality. The use of this document
>> require authorization by the issuer, subject to penalties.
>>
>>
--
Daniel Biazus
infrastructure Engineering
Azion Technologies
Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
Miami, USA +1 305 704 8816
Quaisquer informações contidas neste e-mail e anexos podem ser
confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
de utilização deste documento depende de autorização do emissor, sujeito as
penalidades cabíveis.
Any information in this e-mail and attachments may be confidential and
privileged, protected by legal confidentiality. The use of this document
require authorization by the issuer, subject to penalties.
Re: CPU Usage Analysis
Posted by "Adam W. Dace" <co...@gmail.com>.
If TrafficServer does seem to be spending a lot of CPU time simply writing
data to the network,
one thing some people have had luck with in the past is simply to enable
application-level buffering.
See the documentation, the settings for that are:
proxy.config.net.sock_recv_buffer_size_in
proxy.config.net.sock_send_buffer_size_in
proxy.config.net.sock_recv_buffer_size_out
proxy.config.net.sock_send_buffer_size_out
Hope this helps.
Regards,
Adam
On Wed Jan 07 2015 at 6:11:53 AM Daniel Biazus <da...@azion.com>
wrote:
> Hi Guys,
>
> We have been facing a very similar issue. I got this with with perf top
> tool:
>
> 88.18% traffic_server [.] _Z15write_to_net_
> ioP10NetHandlerP18UnixNetVConnectionP7EThread
> 0.32% traffic_server [.] _ZN10NetHandler12mainNetEventEiP5Event
> 0.30% [kernel] [k] update_sd_lb_stats
> 0.29% [e1000e] [k] e1000e_check_ltr_demote
> 0.25% [kernel] [k] __ticket_spin_lock
> 0.24% traffic_server [.] _ZN7EThread13process_eventEP5Eventi
> 0.21% [kernel] [k] timerqueue_add
> 0.17% libc-2.12.so [.] epoll_wait
> 0.17% libpcre.so.0.0.1 [.] 0x00000000000100dd
> 0.14% [kernel] [k] __schedule
>
> Any thoughts ?
>
> Best Regards,
> Daniel Biazus
>
>
> On Tue, Jan 6, 2015 at 10:36 PM, Jason J. W. Williams <
> jasonjwwilliams@gmail.com> wrote:
>
>> Thanks Sudheer. I was thinking maybe there was some built-in
>> instrumentation that would help. During the spikes all perf shows me
>> as the top calls are:
>>
>> 9.79% [kernel] [k] _raw_spin_lock
>> 9.44% [kernel] [k] find_busiest_group
>>
>> Which isn't very helpful.
>>
>> -J
>>
>
>
>
> --
>
> Daniel Biazus
> infrastructure Engineering
> Azion Technologies
> Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
> Miami, USA +1 305 704 8816
>
> Quaisquer informações contidas neste e-mail e anexos podem ser
> confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
> de utilização deste documento depende de autorização do emissor, sujeito as
> penalidades cabíveis.
>
> Any information in this e-mail and attachments may be confidential and
> privileged, protected by legal confidentiality. The use of this document
> require authorization by the issuer, subject to penalties.
>
>
Re: CPU Usage Analysis
Posted by Daniel Biazus <da...@azion.com>.
Hi Guys,
We have been facing a very similar issue. I got this with with perf top
tool:
88.18% traffic_server [.]
_Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread
0.32% traffic_server [.] _ZN10NetHandler12mainNetEventEiP5Event
0.30% [kernel] [k] update_sd_lb_stats
0.29% [e1000e] [k] e1000e_check_ltr_demote
0.25% [kernel] [k] __ticket_spin_lock
0.24% traffic_server [.] _ZN7EThread13process_eventEP5Eventi
0.21% [kernel] [k] timerqueue_add
0.17% libc-2.12.so [.] epoll_wait
0.17% libpcre.so.0.0.1 [.] 0x00000000000100dd
0.14% [kernel] [k] __schedule
Any thoughts ?
Best Regards,
Daniel Biazus
On Tue, Jan 6, 2015 at 10:36 PM, Jason J. W. Williams <
jasonjwwilliams@gmail.com> wrote:
> Thanks Sudheer. I was thinking maybe there was some built-in
> instrumentation that would help. During the spikes all perf shows me
> as the top calls are:
>
> 9.79% [kernel] [k] _raw_spin_lock
> 9.44% [kernel] [k] find_busiest_group
>
> Which isn't very helpful.
>
> -J
>
--
Daniel Biazus
infrastructure Engineering
Azion Technologies
Porto Alegre, Brasil +55 51 3012 3005 | +55 51 82279032
Miami, USA +1 305 704 8816
Quaisquer informações contidas neste e-mail e anexos podem ser
confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
de utilização deste documento depende de autorização do emissor, sujeito as
penalidades cabíveis.
Any information in this e-mail and attachments may be confidential and
privileged, protected by legal confidentiality. The use of this document
require authorization by the issuer, subject to penalties.
Re: CPU Usage Analysis
Posted by "Jason J. W. Williams" <ja...@gmail.com>.
Thanks Sudheer. I was thinking maybe there was some built-in
instrumentation that would help. During the spikes all perf shows me
as the top calls are:
9.79% [kernel] [k] _raw_spin_lock
9.44% [kernel] [k] find_busiest_group
Which isn't very helpful.
-J
Re: CPU Usage Analysis
Posted by Sudheer Vinukonda <su...@yahoo-inc.com>.
I use the command "perf top -p $(pidof traffic_server)" on RHEL, to understand CPU usage.
Thanks,
Sudheer
On Tuesday, January 6, 2015 3:13 PM, Jason J. W. Williams <ja...@gmail.com> wrote:
I'm seeing CPU spikes sporadically and would like to figure out what
activities (caching, network transfer, etc) in ATS are using the most
CPU so I can tune them. Is there a good way to analyze that?
-J