You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "Sudheer Vinukonda (JIRA)" <ji...@apache.org> on 2014/10/02 16:29:34 UTC
[jira] [Comment Edited] (TS-3102) Reuse context memory for SPDY
streams within a SPDY connection
[ https://issues.apache.org/jira/browse/TS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156610#comment-14156610 ]
Sudheer Vinukonda edited comment on TS-3102 at 10/2/14 2:28 PM:
----------------------------------------------------------------
Below is the dump for various memory allocators with SPDY on (975 qps, 26759 client conns)
proxy.process.spdy.current_client_sessions 12800
{code}
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
67108864 | 0 | 2097152 | memory/ioBufAllocator[14]
134217728 | 68157440 | 1048576 | memory/ioBufAllocator[13]
83886080 | 52428800 | 524288 | memory/ioBufAllocator[12]
125829120 | 96206848 | 262144 | memory/ioBufAllocator[11]
155189248 | 123863040 | 131072 | memory/ioBufAllocator[10]
77594624 | 65798144 | 65536 | memory/ioBufAllocator[9]
4507828224 | 975634432 | 32768 | memory/ioBufAllocator[8]
473432064 | 471482368 | 16384 | memory/ioBufAllocator[7]
1114374144 | 289726464 | 8192 | memory/ioBufAllocator[6]
1307574272 | 410095616 | 4096 | memory/ioBufAllocator[5]
3670016 | 3557376 | 2048 | memory/ioBufAllocator[4]
1966080 | 1818624 | 1024 | memory/ioBufAllocator[3]
917504 | 872448 | 512 | memory/ioBufAllocator[2]
66551808 | 2129920 | 256 | memory/ioBufAllocator[1]
98304 | 77056 | 128 | memory/ioBufAllocator[0]
59375616 | 7501824 | 512 | memory/FetchSMAllocator
0 | 0 | 112 | memory/ICPPeerReadContAllocator
0 | 0 | 432 | memory/PeerReadDataAllocator
0 | 0 | 240 | memory/INKVConnAllocator
1351680 | 639552 | 96 | memory/INKContAllocator
1486848 | 839296 | 32 | memory/apiHookAllocator
0 | 0 | 96 | memory/prefetchLockHandlerAllocator
0 | 0 | 320 | memory/PrefetchBlasterAllocator
0 | 0 | 192 | memory/prefetchUrlEntryAllocator
0 | 0 | 128 | memory/socksProxyAllocator
29417472 | 16528512 | 672 | memory/httpClientSessionAllocator
91193344 | 50746432 | 7744 | memory/httpSMAllocator
2179072 | 1508640 | 224 | memory/httpServerSessionAllocator
0 | 0 | 48 | memory/CacheLookupHttpConfigAllocator
0 | 0 | 7808 | memory/httpUpdateSMAllocator
25976832 | 3281376 | 224 | memory/spdyRequestAllocator
3887104 | 2285504 | 208 | memory/spdyClientSessionAllocator
0 | 0 | 48 | memory/CongestRequestParamAllocator
0 | 0 | 144 | memory/CongestionDBContAllocator
32768 | 0 | 256 | memory/httpCacheAltAllocator
0 | 0 | 112 | memory/OneWayTunnelAllocator
589824 | 2304 | 2304 | memory/hostDBContAllocator
270848 | 33856 | 33856 | memory/dnsBufAllocator
163840 | 0 | 1280 | memory/dnsEntryAllocator
0 | 0 | 16 | memory/DNSRequestDataAllocator
0 | 0 | 576 | memory/cacheContAllocator
0 | 0 | 112 | memory/inControlAllocator
0 | 0 | 112 | memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 | memory/clusterVCAllocator
28262400 | 17633088 | 736 | memory/sslNetVCAllocator
16732160 | 14234032 | 688 | memory/netVCAllocator
0 | 0 | 128 | memory/udpReadContAllocator
0 | 0 | 160 | memory/udpPacketAllocator
0 | 0 | 384 | memory/socksAllocator
0 | 0 | 128 | memory/UDPIOEventAllocator
94707712 | 12352704 | 64 | memory/ioBlockAllocator
42289152 | 10490064 | 48 | memory/ioDataAllocator
158699520 | 36232560 | 240 | memory/ioAllocator
16998400 | 7003360 | 80 | memory/mutexAllocator
7249920 | 3484032 | 96 | memory/eventAllocator
{code}
the biggest culprits that seem to be reused quite poorly in the above dump are:
32K iobuffers ---> allocated in FetchSM as resp_buffer, and pluginVC as a_to_p_buffer and p_to_abuffer
4K iobuffers --> allocated in FetchSM as req_buffer
----> FetchSM allocator
----> Http Client sessions for SPDY streams are not reused, due to the way PluginVC is implemented
---> HttpSM objects for SPDY streams are also not reused, due to the way PluginVC is implemented
---> SPDY request streams are not reused
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
4507828224 | 975634432 | 32768 | memory/ioBufAllocator[8]
1307574272 | 410095616 | 4096 | memory/ioBufAllocator[5] -
59375616 | 7501824 | 512 | memory/FetchSMAllocator
29417472 | 16528512 | 672 | memory/httpClientSessionAllocator
91193344 | 50746432 | 7744 | memory/httpSMAllocator
25976832 | 3281376 | 224 | memory/spdyRequestAllocator
Below are the corresponding numbers with my reuse patch (808.6 qps 39391 client connections) :
proxy.process.spdy.current_client_sessions 11841
{code}
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
67108864 | 0 | 2097152 | memory/ioBufAllocator[14]
100663296 | 48234496 | 1048576 | memory/ioBufAllocator[13]
67108864 | 49283072 | 524288 | memory/ioBufAllocator[12]
117440512 | 103022592 | 262144 | memory/ioBufAllocator[11]
155189248 | 121634816 | 131072 | memory/ioBufAllocator[10]
69206016 | 61276160 | 65536 | memory/ioBufAllocator[9]
751828992 | 659357696 | 32768 | memory/ioBufAllocator[8]
389545984 | 386580480 | 16384 | memory/ioBufAllocator[7]
283639808 | 266133504 | 8192 | memory/ioBufAllocator[6]
531628032 | 501116928 | 4096 | memory/ioBufAllocator[5]
1048576 | 665600 | 2048 | memory/ioBufAllocator[4]
393216 | 293888 | 1024 | memory/ioBufAllocator[3]
131072 | 118272 | 512 | memory/ioBufAllocator[2]
1605632 | 1268736 | 256 | memory/ioBufAllocator[1]
32768 | 14976 | 128 | memory/ioBufAllocator[0]
13631488 | 12464128 | 512 | memory/FetchSMAllocator
0 | 0 | 112 | memory/ICPPeerReadContAllocator
0 | 0 | 432 | memory/PeerReadDataAllocator
0 | 0 | 240 | memory/INKVConnAllocator
626688 | 72480 | 96 | memory/INKContAllocator
806912 | 83328 | 32 | memory/apiHookAllocator
0 | 0 | 96 | memory/prefetchLockHandlerAllocator
0 | 0 | 320 | memory/PrefetchBlasterAllocator
0 | 0 | 192 | memory/prefetchUrlEntryAllocator
0 | 0 | 128 | memory/socksProxyAllocator
30535680 | 26612544 | 672 | memory/httpClientSessionAllocator
56500224 | 5064576 | 7744 | memory/httpSMAllocator
1433600 | 174720 | 224 | memory/httpServerSessionAllocator
0 | 0 | 48 | memory/CacheLookupHttpConfigAllocator
0 | 0 | 7808 | memory/httpUpdateSMAllocator
5963776 | 5453056 | 224 | memory/spdyRequestAllocator
3354624 | 3210624 | 288 | memory/spdyClientSessionAllocator
0 | 0 | 48 | memory/CongestRequestParamAllocator
0 | 0 | 144 | memory/CongestionDBContAllocator
32768 | 256 | 256 | memory/httpCacheAltAllocator
0 | 0 | 112 | memory/OneWayTunnelAllocator
294912 | 2304 | 2304 | memory/hostDBContAllocator
203136 | 33856 | 33856 | memory/dnsBufAllocator
163840 | 0 | 1280 | memory/dnsEntryAllocator
0 | 0 | 16 | memory/DNSRequestDataAllocator
0 | 0 | 576 | memory/cacheContAllocator
0 | 0 | 112 | memory/inControlAllocator
0 | 0 | 112 | memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 | memory/clusterVCAllocator
16109568 | 15902016 | 736 | memory/sslNetVCAllocator
13826048 | 9618240 | 688 | memory/netVCAllocator
0 | 0 | 128 | memory/udpReadContAllocator
0 | 0 | 160 | memory/udpPacketAllocator
0 | 0 | 384 | memory/socksAllocator
0 | 0 | 128 | memory/UDPIOEventAllocator
14745600 | 13494144 | 64 | memory/ioBlockAllocator
10899456 | 10383648 | 48 | memory/ioDataAllocator
45772800 | 41535120 | 240 | memory/ioAllocator
7628800 | 7021040 | 80 | memory/mutexAllocator
3268608 | 2921472 | 96 | memory/eventAllocator
{code}
Below are the corresponding SPDY related buffers that are now much better reused
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
751828992 | 659357696 | 32768 | memory/ioBufAllocator[8]
531628032 | 501116928 | 4096 | memory/ioBufAllocator[5]
13631488 | 12464128 | 512 | memory/FetchSMAllocator
30535680 | 26612544 | 672 | memory/httpClientSessionAllocator
56500224 | 5064576 | 7744 | memory/httpSMAllocator
5963776 | 5453056 | 224 | memory/spdyRequestAllocator
was (Author: sudheerv):
Below is the dump for various memory allocators with SPDY on (975 qps, 26759 client conns)
proxy.process.spdy.current_client_sessions 12800
{code}
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
67108864 | 0 | 2097152 | memory/ioBufAllocator[14]
134217728 | 68157440 | 1048576 | memory/ioBufAllocator[13]
83886080 | 52428800 | 524288 | memory/ioBufAllocator[12]
125829120 | 96206848 | 262144 | memory/ioBufAllocator[11]
155189248 | 123863040 | 131072 | memory/ioBufAllocator[10]
77594624 | 65798144 | 65536 | memory/ioBufAllocator[9]
4507828224 | 975634432 | 32768 | memory/ioBufAllocator[8]
473432064 | 471482368 | 16384 | memory/ioBufAllocator[7]
1114374144 | 289726464 | 8192 | memory/ioBufAllocator[6]
1307574272 | 410095616 | 4096 | memory/ioBufAllocator[5]
3670016 | 3557376 | 2048 | memory/ioBufAllocator[4]
1966080 | 1818624 | 1024 | memory/ioBufAllocator[3]
917504 | 872448 | 512 | memory/ioBufAllocator[2]
66551808 | 2129920 | 256 | memory/ioBufAllocator[1]
98304 | 77056 | 128 | memory/ioBufAllocator[0]
59375616 | 7501824 | 512 | memory/FetchSMAllocator
0 | 0 | 112 | memory/ICPPeerReadContAllocator
0 | 0 | 432 | memory/PeerReadDataAllocator
0 | 0 | 240 | memory/INKVConnAllocator
1351680 | 639552 | 96 | memory/INKContAllocator
1486848 | 839296 | 32 | memory/apiHookAllocator
0 | 0 | 96 | memory/prefetchLockHandlerAllocator
0 | 0 | 320 | memory/PrefetchBlasterAllocator
0 | 0 | 192 | memory/prefetchUrlEntryAllocator
0 | 0 | 128 | memory/socksProxyAllocator
29417472 | 16528512 | 672 | memory/httpClientSessionAllocator
91193344 | 50746432 | 7744 | memory/httpSMAllocator
2179072 | 1508640 | 224 | memory/httpServerSessionAllocator
0 | 0 | 48 | memory/CacheLookupHttpConfigAllocator
0 | 0 | 7808 | memory/httpUpdateSMAllocator
25976832 | 3281376 | 224 | memory/spdyRequestAllocator
3887104 | 2285504 | 208 | memory/spdyClientSessionAllocator
0 | 0 | 48 | memory/CongestRequestParamAllocator
0 | 0 | 144 | memory/CongestionDBContAllocator
32768 | 0 | 256 | memory/httpCacheAltAllocator
0 | 0 | 112 | memory/OneWayTunnelAllocator
589824 | 2304 | 2304 | memory/hostDBContAllocator
270848 | 33856 | 33856 | memory/dnsBufAllocator
163840 | 0 | 1280 | memory/dnsEntryAllocator
0 | 0 | 16 | memory/DNSRequestDataAllocator
0 | 0 | 576 | memory/cacheContAllocator
0 | 0 | 112 | memory/inControlAllocator
0 | 0 | 112 | memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 | memory/clusterVCAllocator
28262400 | 17633088 | 736 | memory/sslNetVCAllocator
16732160 | 14234032 | 688 | memory/netVCAllocator
0 | 0 | 128 | memory/udpReadContAllocator
0 | 0 | 160 | memory/udpPacketAllocator
0 | 0 | 384 | memory/socksAllocator
0 | 0 | 128 | memory/UDPIOEventAllocator
94707712 | 12352704 | 64 | memory/ioBlockAllocator
42289152 | 10490064 | 48 | memory/ioDataAllocator
158699520 | 36232560 | 240 | memory/ioAllocator
16998400 | 7003360 | 80 | memory/mutexAllocator
7249920 | 3484032 | 96 | memory/eventAllocator
{code}
the biggest culprits that seem to be reused quite poorly in the above dump are:
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
4507828224 | 975634432 | 32768 | memory/ioBufAllocator[8] ---> allocated in FetchSM as resp_buffer, and pluginVC as a_to_p_buffer and p_to_abuffer
1307574272 | 410095616 | 4096 | memory/ioBufAllocator[5] ---> allocated in FetchSM as req_buffer
59375616 | 7501824 | 512 | memory/FetchSMAllocator ----> FetchSM allocator
29417472 | 16528512 | 672 | memory/httpClientSessionAllocator ----> Http Client sessions for SPDY streams are not reused, due to the way PluginVC is implemented
91193344 | 50746432 | 7744 | memory/httpSMAllocator ---> HttpSM objects for SPDY streams are also not reused, due to the way PluginVC is implemented
25976832 | 3281376 | 224 | memory/spdyRequestAllocator ---> SPDY request streams are not reused
Below are the corresponding numbers with my reuse patch (808.6 qps 39391 client connections) :
proxy.process.spdy.current_client_sessions 11841
{code}
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
67108864 | 0 | 2097152 | memory/ioBufAllocator[14]
100663296 | 48234496 | 1048576 | memory/ioBufAllocator[13]
67108864 | 49283072 | 524288 | memory/ioBufAllocator[12]
117440512 | 103022592 | 262144 | memory/ioBufAllocator[11]
155189248 | 121634816 | 131072 | memory/ioBufAllocator[10]
69206016 | 61276160 | 65536 | memory/ioBufAllocator[9]
751828992 | 659357696 | 32768 | memory/ioBufAllocator[8]
389545984 | 386580480 | 16384 | memory/ioBufAllocator[7]
283639808 | 266133504 | 8192 | memory/ioBufAllocator[6]
531628032 | 501116928 | 4096 | memory/ioBufAllocator[5]
1048576 | 665600 | 2048 | memory/ioBufAllocator[4]
393216 | 293888 | 1024 | memory/ioBufAllocator[3]
131072 | 118272 | 512 | memory/ioBufAllocator[2]
1605632 | 1268736 | 256 | memory/ioBufAllocator[1]
32768 | 14976 | 128 | memory/ioBufAllocator[0]
13631488 | 12464128 | 512 | memory/FetchSMAllocator
0 | 0 | 112 | memory/ICPPeerReadContAllocator
0 | 0 | 432 | memory/PeerReadDataAllocator
0 | 0 | 240 | memory/INKVConnAllocator
626688 | 72480 | 96 | memory/INKContAllocator
806912 | 83328 | 32 | memory/apiHookAllocator
0 | 0 | 96 | memory/prefetchLockHandlerAllocator
0 | 0 | 320 | memory/PrefetchBlasterAllocator
0 | 0 | 192 | memory/prefetchUrlEntryAllocator
0 | 0 | 128 | memory/socksProxyAllocator
30535680 | 26612544 | 672 | memory/httpClientSessionAllocator
56500224 | 5064576 | 7744 | memory/httpSMAllocator
1433600 | 174720 | 224 | memory/httpServerSessionAllocator
0 | 0 | 48 | memory/CacheLookupHttpConfigAllocator
0 | 0 | 7808 | memory/httpUpdateSMAllocator
5963776 | 5453056 | 224 | memory/spdyRequestAllocator
3354624 | 3210624 | 288 | memory/spdyClientSessionAllocator
0 | 0 | 48 | memory/CongestRequestParamAllocator
0 | 0 | 144 | memory/CongestionDBContAllocator
32768 | 256 | 256 | memory/httpCacheAltAllocator
0 | 0 | 112 | memory/OneWayTunnelAllocator
294912 | 2304 | 2304 | memory/hostDBContAllocator
203136 | 33856 | 33856 | memory/dnsBufAllocator
163840 | 0 | 1280 | memory/dnsEntryAllocator
0 | 0 | 16 | memory/DNSRequestDataAllocator
0 | 0 | 576 | memory/cacheContAllocator
0 | 0 | 112 | memory/inControlAllocator
0 | 0 | 112 | memory/outControlAllocator
0 | 0 | 32 | memory/byteBankAllocator
0 | 0 | 592 | memory/clusterVCAllocator
16109568 | 15902016 | 736 | memory/sslNetVCAllocator
13826048 | 9618240 | 688 | memory/netVCAllocator
0 | 0 | 128 | memory/udpReadContAllocator
0 | 0 | 160 | memory/udpPacketAllocator
0 | 0 | 384 | memory/socksAllocator
0 | 0 | 128 | memory/UDPIOEventAllocator
14745600 | 13494144 | 64 | memory/ioBlockAllocator
10899456 | 10383648 | 48 | memory/ioDataAllocator
45772800 | 41535120 | 240 | memory/ioAllocator
7628800 | 7021040 | 80 | memory/mutexAllocator
3268608 | 2921472 | 96 | memory/eventAllocator
{code}
Below are the corresponding SPDY related buffers that are now much better reused
allocated | in-use | type size | free list name
--------------------|--------------------|------------|----------------------------------
751828992 | 659357696 | 32768 | memory/ioBufAllocator[8]
531628032 | 501116928 | 4096 | memory/ioBufAllocator[5]
13631488 | 12464128 | 512 | memory/FetchSMAllocator
30535680 | 26612544 | 672 | memory/httpClientSessionAllocator
56500224 | 5064576 | 7744 | memory/httpSMAllocator
5963776 | 5453056 | 224 | memory/spdyRequestAllocator
> Reuse context memory for SPDY streams within a SPDY connection
> --------------------------------------------------------------
>
> Key: TS-3102
> URL: https://issues.apache.org/jira/browse/TS-3102
> Project: Traffic Server
> Issue Type: Improvement
> Components: SPDY
> Affects Versions: 5.0.1
> Reporter: Sudheer Vinukonda
> Assignee: Sudheer Vinukonda
> Labels: yahoo
> Fix For: 5.2.0
>
>
> In the present SPDY implementation in ATS, there is no client context reuse. Though the spdy session is reused, each stream (even the non-concurrent ones) is allocated a set of new/separate context buffers (including internal plugin_vc, client_session, SM object, header heap, transaction buffers, server session objects etc). Some of these objects are allocated from global pool, while some are from per-thread pool. The context memory is not reused unlike the non-spdy session, where there can be at most one transaction on a given connection/session at a time.
> Besides being very inefficient (the allocation/deallocation) this also leads to larger memory foot print over time, due to the relatively poor reuse of per thread pool objects (especially, when there are a high number of threads - e.g 100+ like we have).
> I am currently testing a patch that does not deallocate the streams when the transaction completes and reuses those for new/subsequent streams.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)