You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "Sudheer Vinukonda (JIRA)" <ji...@apache.org> on 2014/05/21 02:59:37 UTC
[jira] [Commented] (TS-2824) ioBufferAllocator objects in ATS
master keep growing endlessly
[ https://issues.apache.org/jira/browse/TS-2824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14004174#comment-14004174 ]
Sudheer Vinukonda commented on TS-2824:
---------------------------------------
Update - ATS with TS-2592 reverted looks much better with respect to memory - ioBuf's don't grow anymore and stay at a steady state. An example below
{code}
939524096 | 899678208 | 1048576 | memory/ioBufAllocator[13]
939524096 | 929038336 | 1048576 | memory/ioBufAllocator[13]
939524096 | 913309696 | 1048576 | memory/ioBufAllocator[13]
939524096 | 927989760 | 1048576 | memory/ioBufAllocator[13]
939524096 | 907018240 | 1048576 | memory/ioBufAllocator[13]
973078528 | 936378368 | 1048576 | memory/ioBufAllocator[13]
973078528 | 899678208 | 1048576 | memory/ioBufAllocator[13]
973078528 | 888143872 | 1048576 | memory/ioBufAllocator[13]
973078528 | 903872512 | 1048576 | memory/ioBufAllocator[13]
973078528 | 816840704 | 1048576 | memory/ioBufAllocator[13]
973078528 | 831520768 | 1048576 | memory/ioBufAllocator[13]
973078528 | 834666496 | 1048576 | memory/ioBufAllocator[13]
973078528 | 883949568 | 1048576 | memory/ioBufAllocator[13]
973078528 | 903872512 | 1048576 | memory/ioBufAllocator[13]
973078528 | 922746880 | 1048576 | memory/ioBufAllocator[13]
973078528 | 932184064 | 1048576 | memory/ioBufAllocator[13]
973078528 | 924844032 | 1048576 | memory/ioBufAllocator[13]
973078528 | 923795456 | 1048576 | memory/ioBufAllocator[13]
973078528 | 878706688 | 1048576 | memory/ioBufAllocator[13]
973078528 | 871366656 | 1048576 | memory/ioBufAllocator[13]
973078528 | 868220928 | 1048576 | memory/ioBufAllocator[13]
973078528 | 890241024 | 1048576 | memory/ioBufAllocator[13]
{code}
> ioBufferAllocator objects in ATS master keep growing endlessly
> --------------------------------------------------------------
>
> Key: TS-2824
> URL: https://issues.apache.org/jira/browse/TS-2824
> Project: Traffic Server
> Issue Type: Bug
> Components: Core
> Reporter: Sudheer Vinukonda
>
> During our production testing of ATS master, I noticed that ATS memory keeps growing beyond ram cache size limits. Upon enabling the memory allocation dump, it looks like the ioBufAllocator objects keep growing endlessly. Talking to [~psudaemon] and [~amc] on irc, it looks like the issue might be caused due to NOT in-use proxy allocator (local thread) objects accumulating until they reach the per thread object count limit (250). This may be significant memory for large sized objects (e.g 1MB or 2MB). [~psudaemon] recommended to try reverting the changes in TS-2592 that force to use Proxy allocator instead of global pool objects for ioBuffers. (Note that my production host has ram cache set to 6G and that is completely used, but, the ioBuffers just keep growing)
> Here's the dump of memory allocations:
> {code}
> -bash-4.1$ sudo grep "ioBufAllocator\[13\]" traffic.out
> 0 | 0 | 1048576 | memory/ioBufAllocator[13]
> 1174405120 | 1151336448 | 1048576 | memory/ioBufAllocator[13]
> 4194304000 | 4186963968 | 1048576 | memory/ioBufAllocator[13]
> 1912602624 | 1908408320 | 1048576 | memory/ioBufAllocator[13]
> 771751936 | 759169024 | 1048576 | memory/ioBufAllocator[13]
> 1610612736 | 1608515584 | 1048576 | memory/ioBufAllocator[13]
> 1509949440 | 1484783616 | 1048576 | memory/ioBufAllocator[13]
> 3724541952 | 3697278976 | 1048576 | memory/ioBufAllocator[13]
> 2046820352 | 2023751680 | 1048576 | memory/ioBufAllocator[13]
> 2046820352 | 2036334592 | 1048576 | memory/ioBufAllocator[13]
> 2147483648 | 2128609280 | 1048576 | memory/ioBufAllocator[13]
> 2181038080 | 2172649472 | 1048576 | memory/ioBufAllocator[13]
> 2214592512 | 2204106752 | 1048576 | memory/ioBufAllocator[13]
> 2281701376 | 2253389824 | 1048576 | memory/ioBufAllocator[13]
> 2315255808 | 2287992832 | 1048576 | memory/ioBufAllocator[13]
> 2348810240 | 2316304384 | 1048576 | memory/ioBufAllocator[13]
> 2348810240 | 2336227328 | 1048576 | memory/ioBufAllocator[13]
> 2382364672 | 2380267520 | 1048576 | memory/ioBufAllocator[13]
> 2449473536 | 2424307712 | 1048576 | memory/ioBufAllocator[13]
> 2449473536 | 2447376384 | 1048576 | memory/ioBufAllocator[13]
> 2483027968 | 2474639360 | 1048576 | memory/ioBufAllocator[13]
> 2516582400 | 2507145216 | 1048576 | memory/ioBufAllocator[13]
> 2550136832 | 2543845376 | 1048576 | memory/ioBufAllocator[13]
> 2617245696 | 2599419904 | 1048576 | memory/ioBufAllocator[13]
> 2650800128 | 2626682880 | 1048576 | memory/ioBufAllocator[13]
> 2684354560 | 2659188736 | 1048576 | memory/ioBufAllocator[13]
> 2684354560 | 2675965952 | 1048576 | memory/ioBufAllocator[13]
> 2717908992 | 2690646016 | 1048576 | memory/ioBufAllocator[13]
> 2717908992 | 2713714688 | 1048576 | memory/ioBufAllocator[13]
> 2785017856 | 2758803456 | 1048576 | memory/ioBufAllocator[13]
> 2818572288 | 2792357888 | 1048576 | memory/ioBufAllocator[13]
> 2852126720 | 2823815168 | 1048576 | memory/ioBufAllocator[13]
> 2885681152 | 2882535424 | 1048576 | memory/ioBufAllocator[13]
> 2952790016 | 2921332736 | 1048576 | memory/ioBufAllocator[13]
> 2986344448 | 2959081472 | 1048576 | memory/ioBufAllocator[13]
> 2986344448 | 2985295872 | 1048576 | memory/ioBufAllocator[13]
> 3019898880 | 3014656000 | 1048576 | memory/ioBufAllocator[13]
> 3087007744 | 3067084800 | 1048576 | memory/ioBufAllocator[13]
> 3120562176 | 3104833536 | 1048576 | memory/ioBufAllocator[13]
> 3154116608 | 3146776576 | 1048576 | memory/ioBufAllocator[13]
> 3187671040 | 3186622464 | 1048576 | memory/ioBufAllocator[13]
> 3221225472 | 3220176896 | 1048576 | memory/ioBufAllocator[13]
> 3288334336 | 3268411392 | 1048576 | memory/ioBufAllocator[13]
> 3321888768 | 3292528640 | 1048576 | memory/ioBufAllocator[13]
> 3355443200 | 3344957440 | 1048576 | memory/ioBufAllocator[13]
> 3388997632 | 3364880384 | 1048576 | memory/ioBufAllocator[13]
> 3388997632 | 3387949056 | 1048576 | memory/ioBufAllocator[13]
> 3422552064 | 3404726272 | 1048576 | memory/ioBufAllocator[13]
> 3456106496 | 3423600640 | 1048576 | memory/ioBufAllocator[13]
> Complete memory dump at some point:
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 201326592 | 174063616 | 2097152 | memory/ioBufAllocator[14]
> 3623878656 | 3597664256 | 1048576 | memory/ioBufAllocator[13]
> 1811939328 | 1811939328 | 524288 | memory/ioBufAllocator[12]
> 1400897536 | 1400635392 | 262144 | memory/ioBufAllocator[11]
> 1283457024 | 1281490944 | 131072 | memory/ioBufAllocator[10]
> 1663041536 | 1542127616 | 65536 | memory/ioBufAllocator[9]
> 1693450240 | 1058897920 | 32768 | memory/ioBufAllocator[8]
> 725614592 | 635879424 | 16384 | memory/ioBufAllocator[7]
> 398196736 | 357244928 | 8192 | memory/ioBufAllocator[6]
> 856686592 | 833167360 | 4096 | memory/ioBufAllocator[5]
> 0 | 0 | 2048 | memory/ioBufAllocator[4]
> 131072 | 61440 | 1024 | memory/ioBufAllocator[3]
> 65536 | 24576 | 512 | memory/ioBufAllocator[2]
> 1638400 | 1135616 | 256 | memory/ioBufAllocator[1]
> 1081344 | 643072 | 128 | memory/ioBufAllocator[0]
> 4325376 | 250368 | 512 | memory/FetchSMAllocator
> 0 | 0 | 592 | memory/ICPRequestCont_allocator
> 0 | 0 | 112 | memory/ICPPeerReadContAllocator
> 0 | 0 | 432 | memory/PeerReadDataAllocator
> 4096 | 0 | 32 | memory/MIMEFieldSDKHandle
> 0 | 0 | 240 | memory/INKVConnAllocator
> 12288 | 1248 | 96 | memory/INKContAllocator
> 4096 | 32 | 32 | memory/apiHookAllocator
> 0 | 0 | 96 | memory/prefetchLockHandlerAllocator
> 0 | 0 | 320 | memory/PrefetchBlasterAllocator
> 0 | 0 | 208 | memory/prefetchUrlBlaster
> 0 | 0 | 224 | memory/blasterUrlList
> 0 | 0 | 192 | memory/prefetchUrlEntryAllocator
> 0 | 0 | 128 | memory/socksProxyAllocator
> 0 | 0 | 144 | memory/ObjectReloadCont
> 16625664 | 13732048 | 656 | memory/httpClientSessionAllocator
> 56150016 | 1593072 | 7696 | memory/httpSMAllocator
> 200704 | 62720 | 224 | memory/httpServerSessionAllocator
> 0 | 0 | 48 | memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 7744 | memory/httpUpdateSMAllocator
> 1892352 | 109312 | 224 | memory/spdyRequestAllocator
> 1794048 | 1698432 | 192 | memory/spdyClientSessionAllocator
> 0 | 0 | 128 | memory/RemapPluginsAlloc
> 0 | 0 | 48 | memory/CongestRequestParamAllocator
> 0 | 0 | 144 | memory/CongestionDBContAllocator
> 360448 | 1792 | 256 | memory/httpCacheAltAllocator
> 41943040 | 20996096 | 2048 | memory/hdrStrHeap
> 58982400 | 23791616 | 2048 | memory/hdrHeap
> 0 | 0 | 112 | memory/OneWayTunnelAllocator
> 589824 | 2304 | 2304 | memory/hostDBContAllocator
> 203136 | 33856 | 33856 | memory/dnsBufAllocator
> 327680 | 0 | 1280 | memory/dnsEntryAllocator
> 0 | 0 | 16 | memory/DNSRequestDataAllocator
> 0 | 0 | 48 | memory/ClusterVConnectionCache::Entry
> 0 | 0 | 576 | memory/cacheContAllocator
> 0 | 0 | 112 | memory/inControlAllocator
> 0 | 0 | 112 | memory/outControlAllocator
> 0 | 0 | 32 | memory/byteBankAllocator
> 0 | 0 | 592 | memory/clusterVCAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 6144 | 0 | 48 | memory/cacheRemoveCont
> 208896 | 207552 | 96 | memory/evacuationBlock
> 9621504 | 6307616 | 928 | memory/cacheVConnection
> 1474560 | 941280 | 160 | memory/openDirEntry
> 0 | 0 | 96 | memory/RamCacheCLFUSEntry
> 8986624 | 8785728 | 64 | memory/RamCacheLRUEntry
> 69396480 | 68240160 | 720 | memory/sslNetVCAllocator
> 0 | 0 | 128 | memory/UDPIOEventAllocator
> 13473792 | 11788192 | 688 | memory/netVCAllocator
> 0 | 0 | 128 | memory/udpReadContAllocator
> 0 | 0 | 160 | memory/udpPacketAllocator
> 0 | 0 | 384 | memory/socksAllocator
> 14876672 | 12889408 | 64 | memory/ioBlockAllocator
> 17006592 | 15877632 | 48 | memory/ioDataAllocator
> 34836480 | 27895920 | 240 | memory/ioAllocator
> 11970560 | 11386880 | 80 | memory/mutexAllocator
> 3170304 | 1586976 | 96 | memory/eventAllocator
> 4456448 | 159744 | 1024 | memory/ArenaBlock
> top command showing traffic_server taking 15g RES mem:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 6274 nobody 20 0 19.9g 15g 8428 S 480.3 65.2 1798:44 /home/y/bin/traffic_server -M --httpport 80:fd=7,81:fd=8,82:fd=9,83:fd=10,84:fd=11,85:fd=12,90:fd=13,80:fd=14:ipv6,81:fd=15
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)