[users@httpd] memory usage appearing to increase with time

[Paul Beckett <pa...@outlook.com>]

I am seeing memory usage, as reported by free creeping up with time, before eventually plateauing. Free reported 2,495Gb used, and 1,337Gb free. The summary in top seemed to concur with these values. However if I add up the memory used by various process I cannot account for anywhere near that. Using ps_mem.pyhttp://www.pixelbeat.org/scripts/ps_mem.py , I can only account for 763Mb of RAM usage.I am taking the free memory value from the -/+ buffers/cache line, so this doesn't seem the widely encountered issue where people sometimes mistakenly read it from the first liine, which includes the buffering+caching:http://www.linuxatemyram.com/This only seems to happen when I run Apache httpd on the server. I'm running a source build of Apache 2.4.3 on Redhat Enterprise Linux 6. Apache is only serving around one page a minute (as the server isn't in production usage yet).To see if the RAM reported as used by free, was actually available for use I used the "C munch program" at:http://www.linuxatemyram.com/play.html to attempt to allocate 3000Mb of memory, far more than free suggested was available, but what I thought should be available based upon the memory accounted for by ps_mem.py.This displaced some stuff into swap. But after the memory allocation program exited, free now reports much lower memory usage, which is far closer to what I would expect based upon ps_mem outputMem + swap before munch 2589Mem + swap after munch 1184If anyone can shed any light as to what's going on with the memory usage I would be very grateful. I have included the output of free, top (sorted by memory usage) and ps_mem.py below.Thanks,Paulfree -mtotal used free shared buffers cachedMem: 3833 2896 937 0 282 118-/+ buffers/cache: 2495 1337Swap: 6142 94 6048top - 09:02:54 up 7 days, 17:22, 1 user, load average: 0.07, 0.02, 0.00Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombieCpu(s): 1.0%us, 0.3%sy, 0.0%ni, 98.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%stMem: 3925316k total, 2966540k used, 958776k free, 289028k buffersSwap: 6290424k total, 97100k used, 6193324k free, 121432k cachedPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND26523 sits_tom 20 0 1372m 209m 6244 S 0.0 5.5 2:00.14 java26485 sits_tom 20 0 1372m 209m 6352 S 0.0 5.5 2:02.43 java26568 sits_tom 20 0 1372m 201m 6248 S 0.0 5.2 1:58.63 java1465 daemon 20 0 85092 41m 3460 S 0.0 1.1 19:15.98 .vasd1600 daemon 20 0 73428 30m 5612 S 0.0 0.8 0:02.31 .vasd1478 daemon 20 0 71312 28m 4164 S 0.0 0.7 41:37.49 .vasd1479 daemon 20 0 88908 18m 5684 S 0.0 0.5 8:07.72 .vasd5889 web_apac 20 0 1658m 17m 3864 S 1.0 0.5 12:50.66 httpd1455 root 20 0 53060 9640 4892 S 0.0 0.2 1:02.75 .vasd15010 root 20 0 806m 9484 3008 S 0.0 0.2 1:40.46 scxcimprovagt14977 root 20 0 706m 5788 2948 S 0.0 0.1 1:28.11 scxcimserver14845 root 20 0 110m 5784 4728 S 0.0 0.1 0:00.01 sshd14878 root 20 0 184m 4492 3716 S 0.0 0.1 0:00.00 sudo19308 postfix 20 0 87980 3736 2844 S 0.0 0.1 0:00.02 qmgr14652 postfix 20 0 87800 3596 2748 S 0.0 0.1 0:00.00 pickup15044 scom_msa 20 0 341m 3528 2956 S 0.0 0.1 0:06.22 scxcimprovagt19293 root 20 0 78684 3300 2448 S 0.0 0.1 0:00.10 master1319 root 20 0 249m 2668 800 S 0.0 0.1 0:01.44 rsyslogd5879 root 20 0 122m 2536 2144 S 0.0 0.1 0:03.13 httpd14850 s167 20 0 110m 2396 1320 S 0.0 0.1 0:00.01 sshd14851 s167 20 0 121m 2276 1856 S 0.0 0.1 0:00.00 bash14882 root 20 0 105m 2020 1548 S 0.0 0.1 0:00.01 bashps_mem.pyPrivate + Shared = RAM used Program4.0 KiB + 39.5 KiB = 43.5 KiB rpc.idmapd4.0 KiB + 40.5 KiB = 44.5 KiB acpid4.0 KiB + 54.0 KiB = 58.0 KiB rpc.statd0.0 KiB + 71.5 KiB = 71.5 KiB udevd (3)60.0 KiB + 46.5 KiB = 106.5 KiB rpcbind80.0 KiB + 60.0 KiB = 140.0 KiB rhsmcertd (2)24.0 KiB + 123.0 KiB = 147.0 KiB mingetty (6)184.0 KiB + 37.5 KiB = 221.5 KiB auditd256.0 KiB + 81.5 KiB = 337.5 KiB init296.0 KiB + 91.0 KiB = 387.0 KiB ntpd368.0 KiB + 39.5 KiB = 407.5 KiB crond296.0 KiB + 114.5 KiB = 410.5 KiB cronolog (5)532.0 KiB + 211.5 KiB = 743.5 KiB catalina.sh (3)876.0 KiB + 87.5 KiB = 963.5 KiB vmtoolsd976.0 KiB + 421.5 KiB = 1.4 MiB master1.0 MiB + 443.5 KiB = 1.4 MiB pickup1.1 MiB + 446.5 KiB = 1.5 MiB qmgr924.0 KiB + 685.5 KiB = 1.6 MiB bash (2)1.3 MiB + 633.0 KiB = 1.9 MiB sudo2.0 MiB + 59.5 KiB = 2.1 MiB rsyslogd1.3 MiB + 2.1 MiB = 3.4 MiB sshd (3)3.7 MiB + 494.0 KiB = 4.2 MiB scxcimserver7.6 MiB + 1.4 MiB = 9.1 MiB scxcimprovagt (2)14.2 MiB + 2.1 MiB = 16.4 MiB httpd (2)105.2 MiB + 4.7 MiB = 109.9 MiB .vasd (5)601.8 MiB + 5.0 MiB = 606.8 MiB java (3)---------------------------------763.6 MiB=================================### DURING MUNCH - 3000Mb allocatedfree -mtotal used free shared buffers cachedMem: 3833 3714 119 0 4 16-/+ buffers/cache: 3694 139Swap: 6142 508 5634### AFTER MUNCHfree -mtotal used free shared buffers cachedMem: 3833 724 3108 0 5 16-/+ buffers/cache: 703 3130Swap: 6142 481 5661top - 09:05:49 up 7 days, 17:25, 2 users, load average: 0.19, 0.08, 0.03Tasks: 129 total, 1 running, 128 sleeping, 0 stopped, 0 zombieCpu(s): 4.8%us, 0.0%sy, 0.0%ni, 95.2%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%stMem: 3925316k total, 769064k used, 3156252k free, 9128k buffersSwap: 6290424k total, 490536k used, 5799888k free, 36352k cachedPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND26485 sits_tom 20 0 1372m 144m 4380 S 0.0 3.8 2:02.48 java26568 sits_tom 20 0 1372m 61m 4328 S 0.0 1.6 1:58.69 java26523 sits_tom 20 0 1372m 59m 4328 S 0.0 1.6 2:00.20 java1465 daemon 20 0 85092 40m 2908 S 0.0 1.1 19:16.01 .vasd1478 daemon 20 0 71312 25m 3104 S 0.0 0.7 41:37.55 .vasd1600 daemon 20 0 73428 20m 3840 S 0.0 0.5 0:02.31 .vasd1479 daemon 20 0 88908 12m 3832 S 0.0 0.3 8:07.79 .vasd5889 web_apac 20 0 1658m 11m 3528 S 3.2 0.3 12:52.25 httpd1455 root 20 0 53060 7696 3952 S 0.0 0.2 1:02.76 .vasd15010 root 20 0 806m 7188 1696 S 0.0 0.2 1:40.49 scxcimprovagt15108 root 20 0 110m 4616 4588 S 0.0 0.1 0:00.01 sshd14845 root 20 0 110m 4592 4588 S 0.0 0.1 0:00.01 sshd14878 root 20 0 184m 3664 3660 S 0.0 0.1 0:00.00 sudo15141 root 20 0 184m 3660 3656 S 0.0 0.1 0:00.01 sudo19308 postfix 20 0 87980 2796 2792 S 0.0 0.1 0:00.02 qmgr14652 postfix 20 0 87800 2744 2684 S 0.0 0.1 0:00.00 pickup19293 root 20 0 78684 2512 2432 S 0.0 0.1 0:00.10 master5879 root 20 0 122m 2368 2084 S 0.0 0.1 0:03.14 httpd1319 root 20 0 249m 2252 792 S 0.0 0.1 0:01.44 rsyslogdps_mem.pyPrivate + Shared = RAM used Program4.0 KiB + 37.5 KiB = 41.5 KiB rpc.idmapd4.0 KiB + 39.5 KiB = 43.5 KiB acpid4.0 KiB + 40.0 KiB = 44.0 KiB rhsmcertd (2)4.0 KiB + 51.0 KiB = 55.0 KiB rpc.statd4.0 KiB + 68.5 KiB = 72.5 KiB udevd (3)52.0 KiB + 41.5 KiB = 93.5 KiB rpcbind24.0 KiB + 117.0 KiB = 141.0 KiB mingetty (6)112.0 KiB + 38.5 KiB = 150.5 KiB crond92.0 KiB + 81.0 KiB = 173.0 KiB ntpd12.0 KiB + 163.5 KiB = 175.5 KiB catalina.sh (3)176.0 KiB + 35.5 KiB = 211.5 KiB auditd128.0 KiB + 94.5 KiB = 222.5 KiB cronolog (5)160.0 KiB + 71.5 KiB = 231.5 KiB init160.0 KiB + 389.5 KiB = 549.5 KiB pickup180.0 KiB + 392.5 KiB = 572.5 KiB qmgr204.0 KiB + 369.5 KiB = 573.5 KiB master400.0 KiB + 233.0 KiB = 633.0 KiB scxcimserver896.0 KiB + 74.5 KiB = 970.5 KiB vmtoolsd508.0 KiB + 813.0 KiB = 1.3 MiB bash (4)8.0 KiB + 1.4 MiB = 1.4 MiB sudo (2)1.6 MiB + 56.5 KiB = 1.7 MiB rsyslogd304.0 KiB + 2.5 MiB = 2.8 MiB sshd (5)5.8 MiB + 579.0 KiB = 6.4 MiB scxcimprovagt (2)8.3 MiB + 1.8 MiB = 10.1 MiB httpd (2)88.6 MiB + 2.9 MiB = 91.5 MiB .vasd (5)253.7 MiB + 3.2 MiB = 256.9 MiB java (3)---------------------------------376.8 MiB=================================