You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "jaekyung oh (JIRA)" <ji...@apache.org> on 2013/04/01 11:25:19 UTC
[jira] [Commented] (TS-1006) memory management, cut down memory
waste ?
[ https://issues.apache.org/jira/browse/TS-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618711#comment-13618711 ]
jaekyung oh commented on TS-1006:
---------------------------------
Hi. Yunkai Zhang.
I tried again with "./configure --enable-reclaimable-freelist"
but same happens...
these are the part of traffic.out when i executed "traffic_line -x"
....
360192 | 360192 | 672 | memory/netVCAllocator
0 | 0 | 120 | memory/udpReadContAllocator
0 | 0 | 176 | memory/udpPacketAllocator
0 | 0 | 384 | memory/socksAllocator
0 | 0 | 128 | memory/UDPIOEventAllocator
16256 | 16256 | 64 | memory/ioBlockAllocator
8064 | 8064 | 48 | memory/ioDataAllocator
32480 | 32480 | 232 | memory/ioAllocator
109512 | 109512 | 72 | memory/mutexAllocator
318032 | 318032 | 88 | memory/eventAllocator
268288 | 268288 | 1024 | memory/ArenaBlock
[Apr 1 18:18:35.010] Manager {0x7f25beffd700} NOTE: User has changed config file records.config
[Apr 1 18:18:36.678] Server {0x2b989b06b700} NOTE: cache enabled
[2b989b06b700:01][ink_queue_ext.cc:00577][F] 13.28M t:278 f:274 m:0 avg:0.0 M:4 csbase:256 csize:278 tsize:88 cbsize:24576
[2b989b06b700:01][ink_queue_ext.cc:00584][-] 13.28M t:278 f:274 m:0 avg:0.0 M:4 csbase:256 csize:278 tsize:88 cbsize:24576
[2b989b06b700:01][ink_queue_ext.cc:00577][F] 13.32M t:278 f:274 m:0 avg:82.2 M:4 csbase:256 csize:278 tsize:88 cbsize:24576
[2b989b06b700:01][ink_queue_ext.cc:00584][-] 13.32M t:196 f:192 m:0 avg:82.2 M:4 csbase:256 csize:278 tsize:88 cbsize:24576
NOTE: Traffic Server received Sig 11: Segmentation fault <------------------------------------------------ what happened after logging?
/usr/local/nts/bin/traffic_server - STACK TRACE:
/lib64/libpthread.so.0(+0xf2d0)[0x2b9897f1f2d0]
[0x2b989b274f10]
[Apr 1 18:18:39.800] Manager {0x7f25c7ec0720} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
[Apr 1 18:18:39.800] Manager {0x7f25c7ec0720} ERROR: (last system error 2: No such file or directory)
[Apr 1 18:18:39.800] Manager {0x7f25c7ec0720} ERROR: [Alarms::signalAlarm] Server Process was reset
[Apr 1 18:18:39.800] Manager {0x7f25c7ec0720} ERROR: (last system error 2: No such file or directory)
[Apr 1 18:18:40.803] Manager {0x7f25c7ec0720} NOTE: [LocalManager::startProxy] Launching ts process
[TrafficServer] using root directory '/usr/local/nts'
[Apr 1 18:18:40.813] Manager {0x7f25c7ec0720} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '10'
[Apr 1 18:18:40.813] Manager {0x7f25c7ec0720} NOTE: [Alarms::signalAlarm] Server Process born
.....
> memory management, cut down memory waste ?
> ------------------------------------------
>
> Key: TS-1006
> URL: https://issues.apache.org/jira/browse/TS-1006
> Project: Traffic Server
> Issue Type: Improvement
> Components: Core
> Affects Versions: 3.1.1
> Reporter: Zhao Yongming
> Assignee: Bin Chen
> Fix For: 3.3.3
>
> Attachments: 0001-TS-1006-Add-an-enable-reclaimable-freelist-option.patch, 0002-TS-1006-Add-a-new-wrapper-ink_atomic_decrement.patch, 0003-TS-1006-Introduce-a-reclaimable-InkFreeList.patch, 0004-TS-1006-Make-InkFreeList-memory-pool-configurable.patch, Memory-Usage-After-Introduced-New-Allocator.png, memusage.ods, memusage.ods
>
>
> when we review the memory usage in the production, there is something abnormal, ie, looks like TS take much memory than index data + common system waste, and here is some memory dump result by set "proxy.config.dump_mem_info_frequency"
> 1, the one on a not so busy forwarding system:
> physics memory: 32G
> RAM cache: 22G
> DISK: 6140 GB
> average_object_size 64000
> {code}
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 671088640 | 37748736 | 2097152 | memory/ioBufAllocator[14]
> 2248146944 | 2135949312 | 1048576 | memory/ioBufAllocator[13]
> 1711276032 | 1705508864 | 524288 | memory/ioBufAllocator[12]
> 1669332992 | 1667760128 | 262144 | memory/ioBufAllocator[11]
> 2214592512 | 2211840000 | 131072 | memory/ioBufAllocator[10]
> 2325741568 | 2323775488 | 65536 | memory/ioBufAllocator[9]
> 2091909120 | 2089123840 | 32768 | memory/ioBufAllocator[8]
> 1956642816 | 1956478976 | 16384 | memory/ioBufAllocator[7]
> 2094530560 | 2094071808 | 8192 | memory/ioBufAllocator[6]
> 356515840 | 355540992 | 4096 | memory/ioBufAllocator[5]
> 1048576 | 14336 | 2048 | memory/ioBufAllocator[4]
> 131072 | 0 | 1024 | memory/ioBufAllocator[3]
> 65536 | 0 | 512 | memory/ioBufAllocator[2]
> 32768 | 0 | 256 | memory/ioBufAllocator[1]
> 16384 | 0 | 128 | memory/ioBufAllocator[0]
> 0 | 0 | 576 | memory/ICPRequestCont_allocator
> 0 | 0 | 112 | memory/ICPPeerReadContAllocator
> 0 | 0 | 432 | memory/PeerReadDataAllocator
> 0 | 0 | 32 | memory/MIMEFieldSDKHandle
> 0 | 0 | 240 | memory/INKVConnAllocator
> 0 | 0 | 96 | memory/INKContAllocator
> 4096 | 0 | 32 | memory/apiHookAllocator
> 0 | 0 | 288 | memory/FetchSMAllocator
> 0 | 0 | 80 | memory/prefetchLockHandlerAllocator
> 0 | 0 | 176 | memory/PrefetchBlasterAllocator
> 0 | 0 | 80 | memory/prefetchUrlBlaster
> 0 | 0 | 96 | memory/blasterUrlList
> 0 | 0 | 96 | memory/prefetchUrlEntryAllocator
> 0 | 0 | 128 | memory/socksProxyAllocator
> 0 | 0 | 144 | memory/ObjectReloadCont
> 3258368 | 576016 | 592 | memory/httpClientSessionAllocator
> 825344 | 139568 | 208 | memory/httpServerSessionAllocator
> 22597632 | 1284848 | 9808 | memory/httpSMAllocator
> 0 | 0 | 32 | memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 9856 | memory/httpUpdateSMAllocator
> 0 | 0 | 128 | memory/RemapPluginsAlloc
> 0 | 0 | 48 | memory/CongestRequestParamAllocator
> 0 | 0 | 128 | memory/CongestionDBContAllocator
> 5767168 | 704512 | 2048 | memory/hdrStrHeap
> 18350080 | 1153024 | 2048 | memory/hdrHeap
> 53248 | 2912 | 208 | memory/httpCacheAltAllocator
> 0 | 0 | 112 | memory/OneWayTunnelAllocator
> 157696 | 1232 | 1232 | memory/hostDBContAllocator
> 102240 | 17040 | 17040 | memory/dnsBufAllocator
> 323584 | 0 | 1264 | memory/dnsEntryAllocator
> 0 | 0 | 16 | memory/DNSRequestDataAllocator
> 0 | 0 | 1072 | memory/SRVAllocator
> 0 | 0 | 48 | memory/ClusterVConnectionCache::Entry
> 0 | 0 | 560 | memory/cacheContAllocator
> 0 | 0 | 112 | memory/inControlAllocator
> 0 | 0 | 112 | memory/outControlAllocator
> 0 | 0 | 32 | memory/byteBankAllocator
> 0 | 0 | 576 | memory/clusterVCAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 6144 | 0 | 48 | memory/cacheRemoveCont
> 270336 | 262560 | 96 | memory/evacuationBlock
> 4997120 | 3968416 | 976 | memory/cacheVConnection
> 798720 | 522080 | 160 | memory/openDirEntry
> 0 | 0 | 64 | memory/RamCacheLRUEntry
> 56426496 | 56426304 | 96 | memory/RamCacheCLFUSEntry
> 9584640 | 6168000 | 960 | memory/netVCAllocator
> 0 | 0 | 128 | memory/udpReadContAllocator
> 0 | 0 | 128 | memory/udpWorkContinuationAllocator
> 0 | 0 | 160 | memory/udpPacketAllocator
> 0 | 0 | 304 | memory/socksAllocator
> 139264 | 68544 | 1088 | memory/sslNetVCAllocator
> 0 | 0 | 128 | memory/UDPIOEventAllocator
> 671744 | 115520 | 64 | memory/ioBlockAllocator
> 28305408 | 28301520 | 48 | memory/ioDataAllocator
> 2273280 | 406320 | 240 | memory/ioAllocator
> 1904640 | 1489920 | 80 | memory/mutexAllocator
> 1105920 | 188544 | 96 | memory/eventAllocator
> 2359296 | 129024 | 1024 | memory/ArenaBlock
> {code}
> this box will crash every 2days, so the memory waste may no that high
> 2, our production reverse system:
> physics memory: 16G
> RAM cache: 8G
> DISK: 1516 GB
> average_object_size 16384
> and it run for a much long time:
> {code}
> allocated | in-use | type size | free list name
> --------------------|--------------------|------------|----------------------------------
> 805306368 | 0 | 2097152 | memory/ioBufAllocator[14]
> 738197504 | 8388608 | 1048576 | memory/ioBufAllocator[13]
> 1258291200 | 46661632 | 524288 | memory/ioBufAllocator[12]
> 1300234240 | 183762944 | 262144 | memory/ioBufAllocator[11]
> 1170210816 | 466223104 | 131072 | memory/ioBufAllocator[10]
> 1790967808 | 1223426048 | 65536 | memory/ioBufAllocator[9]
> 2970615808 | 2601418752 | 32768 | memory/ioBufAllocator[8]
> 2067791872 | 2044608512 | 16384 | memory/ioBufAllocator[7]
> 1169424384 | 1169121280 | 8192 | memory/ioBufAllocator[6]
> 711458816 | 710463488 | 4096 | memory/ioBufAllocator[5]
> 1572864 | 0 | 2048 | memory/ioBufAllocator[4]
> 131072 | 0 | 1024 | memory/ioBufAllocator[3]
> 65536 | 0 | 512 | memory/ioBufAllocator[2]
> 32768 | 0 | 256 | memory/ioBufAllocator[1]
> 16384 | 0 | 128 | memory/ioBufAllocator[0]
> 0 | 0 | 576 | memory/ICPRequestCont_allocator
> 0 | 0 | 112 | memory/ICPPeerReadContAllocator
> 0 | 0 | 432 | memory/PeerReadDataAllocator
> 0 | 0 | 32 | memory/MIMEFieldSDKHandle
> 0 | 0 | 240 | memory/INKVConnAllocator
> 0 | 0 | 96 | memory/INKContAllocator
> 4096 | 0 | 32 | memory/apiHookAllocator
> 0 | 0 | 288 | memory/FetchSMAllocator
> 0 | 0 | 80 | memory/prefetchLockHandlerAllocator
> 0 | 0 | 176 | memory/PrefetchBlasterAllocator
> 0 | 0 | 80 | memory/prefetchUrlBlaster
> 0 | 0 | 96 | memory/blasterUrlList
> 0 | 0 | 96 | memory/prefetchUrlEntryAllocator
> 0 | 0 | 128 | memory/socksProxyAllocator
> 0 | 0 | 144 | memory/ObjectReloadCont
> 1136640 | 125504 | 592 | memory/httpClientSessionAllocator
> 372736 | 27248 | 208 | memory/httpServerSessionAllocator
> 11317248 | 39296 | 9824 | memory/httpSMAllocator
> 0 | 0 | 32 | memory/CacheLookupHttpConfigAllocator
> 0 | 0 | 9888 | memory/httpUpdateSMAllocator
> 0 | 0 | 128 | memory/RemapPluginsAlloc
> 0 | 0 | 512 | memory/HCSMAllocator
> 0 | 0 | 48 | memory/VCEntryAllocator
> 0 | 0 | 96 | memory/HCEntryAllocator
> 0 | 0 | 64 | memory/HCHandlerAllocator
> 0 | 0 | 48 | memory/CongestRequestParamAllocator
> 0 | 0 | 128 | memory/CongestionDBContAllocator
> 6029312 | 643072 | 2048 | memory/hdrStrHeap
> 7077888 | 657408 | 2048 | memory/hdrHeap
> 26624 | 208 | 208 | memory/httpCacheAltAllocator
> 0 | 0 | 112 | memory/OneWayTunnelAllocator
> 630784 | 1232 | 1232 | memory/hostDBContAllocator
> 238560 | 17040 | 17040 | memory/dnsBufAllocator
> 161792 | 0 | 1264 | memory/dnsEntryAllocator
> 0 | 0 | 16 | memory/DNSRequestDataAllocator
> 0 | 0 | 1072 | memory/SRVAllocator
> 0 | 0 | 48 | memory/ClusterVConnectionCache::Entry
> 0 | 0 | 560 | memory/cacheContAllocator
> 0 | 0 | 112 | memory/inControlAllocator
> 0 | 0 | 112 | memory/outControlAllocator
> 0 | 0 | 32 | memory/byteBankAllocator
> 0 | 0 | 576 | memory/clusterVCAllocator
> 0 | 0 | 48 | memory/evacuationKey
> 6144 | 0 | 48 | memory/cacheRemoveCont
> 17006592 | 14972928 | 96 | memory/evacuationBlock
> 1777664 | 759872 | 992 | memory/cacheVConnection
> 307200 | 111520 | 160 | memory/openDirEntry
> 0 | 0 | 64 | memory/RamCacheLRUEntry
> 104275968 | 104274048 | 96 | memory/RamCacheCLFUSEntry
> 3440640 | 1819200 | 960 | memory/netVCAllocator
> 0 | 0 | 128 | memory/udpReadContAllocator
> 0 | 0 | 128 | memory/udpWorkContinuationAllocator
> 0 | 0 | 160 | memory/udpPacketAllocator
> 0 | 0 | 304 | memory/socksAllocator
> 0 | 0 | 1088 | memory/sslNetVCAllocator
> 0 | 0 | 128 | memory/UDPIOEventAllocator
> 237568 | 22528 | 64 | memory/ioBlockAllocator
> 26087424 | 26081904 | 48 | memory/ioDataAllocator
> 890880 | 84240 | 240 | memory/ioAllocator
> 1525760 | 1403440 | 80 | memory/mutexAllocator
> 565248 | 129696 | 96 | memory/eventAllocator
> 1179648 | 4096 | 1024 | memory/ArenaBlock
> {code}
> our team is working on the memory free issue, trying to improve the memory management. and this a big project, the more input|comment the better.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira