TrafficServer and IO/Wait

[Jean Baptiste Favre <we...@jbfavre.org>]

Hi all,
I'm using Trafficserver 4.2.0 on Debian Wheezy, with raw disks as cache
on VMware virtual machines. Activity level is a bit less than 400
requests per seconds with a cache hit ratio greater than 99%.

On a regular basis, traffic_server process (ET_NET 0) starts to heavily
use CPU (more than 75%).

That's fine for me since it's the only process on the virtual machine.
But, the problem is that most of this CPU usage concerns IO Wait.

Running command iostat -m 1 confirms that all IO Wait operations are
related to cache raw devices.

Restarting trafficserver service fixes the problem... till next time.

I tried to strace the process and got plenty of:
epoll_wait(21, {{EPOLLIN, {u32=301105424, u64=47154747048208}}}, 32768,
10) = 1
read(14, "\1\0\0\0\0\0\0\0", 8)         = 8
futex(0x2ae31800d07c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x2ae31800d078,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x2ae31800d050, FUTEX_WAKE_PRIVATE, 1) = 1

Found nothing strange in diags.log, manager.log or traffic.out log files.

Any idea welcome,

Regards,
Jean-Baptiste Favre

Re: TrafficServer and IO/Wait

[Jan-Frode Myklebust <ja...@tanso.net>]

On Wed, Jun 04, 2014 at 11:28:07AM +0200, Jean Baptiste Favre wrote:
> Hello,
> Each VM has 4GB memory.
> 
> traffic_line -r proxy.config.cache.ram_cache.size gives 2147483648

I would consider increasing proxy.config.cache.ram_cache.size to at
least 3GB on your 4GB VMs. Assuming ATS will have better knowledge about
what to keep in memory than the linux page cache. 

And maybe give the VMs lots of more memory so that you can increase the
ram_cache.size further.

I try to make sure ATS can keep everything in memory -- memory is cheap,
disk IOPS is slow and expensive.



  -jf

Re: TrafficServer and IO/Wait

[Jean Baptiste Favre <we...@jbfavre.org>]

Hello,
Each VM has 4GB memory.

traffic_line -r proxy.config.cache.ram_cache.size gives 2147483648

Cache is splitted on 3 raw disks (sdb, sdc & sdd). Each disk is 10GB and
cache is full:
traffic_line -r proxy.process.cache.percent_full gives 99

I'm not sure how I can find cached entry number.

In terms of client delivery, here are the stats I have:

proxy.node.restarts.manager.start_time: 1401710734
proxy.node.restarts.proxy.start_time: 1401710735

proxy.process.http.response_document_size_100: 1097534
proxy.process.http.response_document_size_1K: 797631771
proxy.process.http.response_document_size_3K: 674981175
proxy.process.http.response_document_size_5K: 149640
proxy.process.http.response_document_size_10K: 27104535
proxy.process.http.response_document_size_1M: 77439445
proxy.process.http.response_document_size_inf: 69

Outgoing network traffic, for each VM, is 22Mbps and stable.

Regards,
Jean-Baptiste Favre

On 04/06/2014 11:02, Jan-Frode Myklebust wrote:
> 
> How much memory does the VM have? How much
> proxy.config.cache.ram_cache.size have you given it? How much data are
> you caching? How much is delivered to clients ?
> 
>   -jf

Re: TrafficServer and IO/Wait

[Jan-Frode Myklebust <ja...@tanso.net>]

How much memory does the VM have? How much
proxy.config.cache.ram_cache.size have you given it? How much data are
you caching? How much is delivered to clients ?



  -jf