You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Kaivan Sanghvi <ka...@gmail.com> on 2016/07/22 17:52:11 UTC
TrafficServer using swap with almost 60GB still available
Hi,
Recently we encountered that the traffic_server process was using about
7GB(total available 8GB) of swap space even when only half of the available
physical memory(~68GB out of 128GB) was used up on the host. Can someone
help us with why this might be happening and if we need to tweak any
configurations for the same?
Here is some more information:
*ATS*: 5.1.2 using 24 6TB HDDs
Ubuntu 14.04.4 LTS
*RAM Cache:*
traffic_line -r proxy.config.cache.ram_cache.size
21474836480
*cat /proc/18038/status*
Name: [ET_NET 0]
State: S (sleeping)
Tgid: 18038
Ngid: 18082
Pid: 18038
PPid: 18017
TracerPid: 0
Uid: 65534 65534 65534 65534
Gid: 65534 65534 65534 65534
FDSize: 1024
Groups: 0
VmPeak: 62507948 kB
VmSize: 60678648 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 55435964 kB
VmRSS: 50802704 kB
VmData: 60601012 kB
VmStk: 136 kB
VmExe: 4188 kB
VmLib: 8624 kB
VmPTE: 105724 kB
*VmSwap: 7145508 kB*
Threads: 247
SigQ: 0/1027883
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000381000
SigCgt: 0000000180004e4f
CapInh: 0000000000000000
CapPrm: 0000000000005402
CapEff: 0000000000005400
CapBnd: 0000001fffffffff
Seccomp: 0
Cpus_allowed: ffffffff
Cpus_allowed_list: 0-31
Mems_allowed: 00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 444748631
nonvoluntary_ctxt_switches: 627214
*HTOP output showing swap usage and almost half of physical memory still
available*:
[image: Inline image 1]
*cat /proc/meminfo*
MemTotal: 131590680 kB
MemFree: 60468340 kB
Buffers: 602848 kB
Cached: 8235876 kB
SwapCached: 2261820 kB
Active: 60177604 kB
Inactive: 8754340 kB
Active(anon): 52968336 kB
Inactive(anon): 7126492 kB
Active(file): 7209268 kB
Inactive(file): 1627848 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 8000508 kB
SwapFree: 4000400 kB
Dirty: 796 kB
Writeback: 0 kB
AnonPages: 59005956 kB
Mapped: 56164 kB
Shmem: 1600 kB
Slab: 1066076 kB
SReclaimable: 835516 kB
SUnreclaim: 230560 kB
KernelStack: 8840 kB
PageTables: 144336 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 73795848 kB
Committed_AS: 64516392 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 484776 kB
VmallocChunk: 34291077428 kB
HardwareCorrupted: 0 kB
AnonHugePages: 39610368 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 113280 kB
DirectMap2M: 3631104 kB
DirectMap1G: 132120576 kB
*cat /proc/sys/vm/max_map_count*
65530
Thanks,
Kaivan.
Re: TrafficServer using swap with almost 60GB still available
Posted by CJ Ess <zx...@gmail.com>.
Does Traffic Server mmap anonymous memory? I believe that would explain
what he is seeing - if the memory isn't backed by a file then its backed by
swap. So in that case the kernel might opt to swap out seldom used portions
of the memory cache in favor of network or disk buffers. If you look at
your disk IO are you swapping heavily?
On Fri, Jul 22, 2016 at 2:20 PM, Jered Floyd <je...@convivian.com> wrote:
>
> Kaivan,
>
> I believe you are misunderstanding how the Linux kernel uses swap. Swap
> is not just used to avoid out-of-memory conditions, but also to free up
> cache memory by evicting infrequently used blocks in memory. The behavior
> you are seeing is not unreasonable, but if you really want to store
> everything in memory (which could reduce performance) you can adjust the
> "swappiness" of your system.
>
> There are many good explanations of this online, here is one:
>
> http://askubuntu.com/questions/157793/why-is-swap-being-used-even-though-i-have-plenty-of-free-ram
>
> --Jered
>
> ----- On Jul 22, 2016, at 1:52 PM, Kaivan Sanghvi <
> kaivan.sanghvi@gmail.com> wrote:
>
> Hi,
>
> Recently we encountered that the traffic_server process was using about
> 7GB(total available 8GB) of swap space even when only half of the available
> physical memory(~68GB out of 128GB) was used up on the host. Can someone
> help us with why this might be happening and if we need to tweak any
> configurations for the same?
>
> Here is some more information:
>
> *ATS*: 5.1.2 using 24 6TB HDDs
>
> Ubuntu 14.04.4 LTS
>
> *RAM Cache:*
> traffic_line -r proxy.config.cache.ram_cache.size
> 21474836480
>
> *cat /proc/18038/status*
>
> Name: [ET_NET 0]
> State: S (sleeping)
> Tgid: 18038
> Ngid: 18082
> Pid: 18038
> PPid: 18017
> TracerPid: 0
> Uid: 65534 65534 65534 65534
> Gid: 65534 65534 65534 65534
> FDSize: 1024
> Groups: 0
> VmPeak: 62507948 kB
> VmSize: 60678648 kB
> VmLck: 0 kB
> VmPin: 0 kB
> VmHWM: 55435964 kB
> VmRSS: 50802704 kB
> VmData: 60601012 kB
> VmStk: 136 kB
> VmExe: 4188 kB
> VmLib: 8624 kB
> VmPTE: 105724 kB
> *VmSwap: 7145508 kB*
> Threads: 247
> SigQ: 0/1027883
> SigPnd: 0000000000000000
> ShdPnd: 0000000000000000
> SigBlk: 0000000000000000
> SigIgn: 0000000000381000
> SigCgt: 0000000180004e4f
> CapInh: 0000000000000000
> CapPrm: 0000000000005402
> CapEff: 0000000000005400
> CapBnd: 0000001fffffffff
> Seccomp: 0
> Cpus_allowed: ffffffff
> Cpus_allowed_list: 0-31
> Mems_allowed: 00000000,00000003
> Mems_allowed_list: 0-1
> voluntary_ctxt_switches: 444748631
> nonvoluntary_ctxt_switches: 627214
>
> *HTOP output showing swap usage and almost half of physical memory still
> available*:
> [image: Inline image 1]
>
> *cat /proc/meminfo*
>
> MemTotal: 131590680 kB
> MemFree: 60468340 kB
> Buffers: 602848 kB
> Cached: 8235876 kB
> SwapCached: 2261820 kB
> Active: 60177604 kB
> Inactive: 8754340 kB
> Active(anon): 52968336 kB
> Inactive(anon): 7126492 kB
> Active(file): 7209268 kB
> Inactive(file): 1627848 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 8000508 kB
> SwapFree: 4000400 kB
> Dirty: 796 kB
> Writeback: 0 kB
> AnonPages: 59005956 kB
> Mapped: 56164 kB
> Shmem: 1600 kB
> Slab: 1066076 kB
> SReclaimable: 835516 kB
> SUnreclaim: 230560 kB
> KernelStack: 8840 kB
> PageTables: 144336 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 73795848 kB
> Committed_AS: 64516392 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 484776 kB
> VmallocChunk: 34291077428 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 39610368 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 113280 kB
> DirectMap2M: 3631104 kB
> DirectMap1G: 132120576 kB
>
> *cat /proc/sys/vm/max_map_count*
> 65530
>
> Thanks,
> Kaivan.
>
>
Re: TrafficServer using swap with almost 60GB still available
Posted by Jered Floyd <je...@convivian.com>.
Kaivan,
I believe you are misunderstanding how the Linux kernel uses swap. Swap is not just used to avoid out-of-memory conditions, but also to free up cache memory by evicting infrequently used blocks in memory. The behavior you are seeing is not unreasonable, but if you really want to store everything in memory (which could reduce performance) you can adjust the "swappiness" of your system.
There are many good explanations of this online, here is one:
http://askubuntu.com/questions/157793/why-is-swap-being-used-even-though-i-have-plenty-of-free-ram
--Jered
----- On Jul 22, 2016, at 1:52 PM, Kaivan Sanghvi <ka...@gmail.com> wrote:
> Hi,
> Recently we encountered that the traffic_server process was using about
> 7GB(total available 8GB) of swap space even when only half of the available
> physical memory(~68GB out of 128GB) was used up on the host. Can someone help
> us with why this might be happening and if we need to tweak any configurations
> for the same?
> Here is some more information:
> ATS : 5.1.2 using 24 6TB HDDs
> Ubuntu 14.04.4 LTS
> RAM Cache:
> traffic_line -r proxy.config.cache.ram_cache.size
> 21474836480
> cat /proc/18038/status
> Name: [ET_NET 0]
> State: S (sleeping)
> Tgid: 18038
> Ngid: 18082
> Pid: 18038
> PPid: 18017
> TracerPid: 0
> Uid: 65534 65534 65534 65534
> Gid: 65534 65534 65534 65534
> FDSize: 1024
> Groups: 0
> VmPeak: 62507948 kB
> VmSize: 60678648 kB
> VmLck: 0 kB
> VmPin: 0 kB
> VmHWM: 55435964 kB
> VmRSS: 50802704 kB
> VmData: 60601012 kB
> VmStk: 136 kB
> VmExe: 4188 kB
> VmLib: 8624 kB
> VmPTE: 105724 kB
> VmSwap: 7145508 kB
> Threads: 247
> SigQ: 0/1027883
> SigPnd: 0000000000000000
> ShdPnd: 0000000000000000
> SigBlk: 0000000000000000
> SigIgn: 0000000000381000
> SigCgt: 0000000180004e4f
> CapInh: 0000000000000000
> CapPrm: 0000000000005402
> CapEff: 0000000000005400
> CapBnd: 0000001fffffffff
> Seccomp: 0
> Cpus_allowed: ffffffff
> Cpus_allowed_list: 0-31
> Mems_allowed: 00000000,00000003
> Mems_allowed_list: 0-1
> voluntary_ctxt_switches: 444748631
> nonvoluntary_ctxt_switches: 627214
> HTOP output showing swap usage and almost half of physical memory still
> available :
> cat /proc/meminfo
> MemTotal: 131590680 kB
> MemFree: 60468340 kB
> Buffers: 602848 kB
> Cached: 8235876 kB
> SwapCached: 2261820 kB
> Active: 60177604 kB
> Inactive: 8754340 kB
> Active(anon): 52968336 kB
> Inactive(anon): 7126492 kB
> Active(file): 7209268 kB
> Inactive(file): 1627848 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 8000508 kB
> SwapFree: 4000400 kB
> Dirty: 796 kB
> Writeback: 0 kB
> AnonPages: 59005956 kB
> Mapped: 56164 kB
> Shmem: 1600 kB
> Slab: 1066076 kB
> SReclaimable: 835516 kB
> SUnreclaim: 230560 kB
> KernelStack: 8840 kB
> PageTables: 144336 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 73795848 kB
> Committed_AS: 64516392 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 484776 kB
> VmallocChunk: 34291077428 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 39610368 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 113280 kB
> DirectMap2M: 3631104 kB
> DirectMap1G: 132120576 kB
> cat /proc/sys/vm/max_map_count
> 65530
> Thanks,
> Kaivan.