You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by GitBox <gi...@apache.org> on 2021/08/19 18:57:36 UTC
[GitHub] [trafficserver] duke8253 opened a new issue #8265: Memory leaks in Http2Stream
duke8253 opened a new issue #8265:
URL: https://github.com/apache/trafficserver/issues/8265
As described in #8260 , we've been observing high memory utilization in production after upgrading to ats9.1. JeMalloc profiling shows the problem is related to Http2Stream.
Here's what I've collected using jemalloc, the graph ignores the memory allocated to iobuffers. After running for 3 hours or so, ats9.1's memory usage is about 6-7GB more than what ats9.0 is using, and this continues to grow till the host eventually runs out of memory.
ats9.0
[n7.pdf](https://github.com/apache/trafficserver/files/7017105/n7.pdf)
ats9.1
[n7.pdf](https://github.com/apache/trafficserver/files/7017104/n7.pdf)
As @shinrich mentioned in the PR:
> Setting zombie event, we found a session that had the http2 inactivity timeout trigger. Http2Stream::initiating_close had been called on the stream, but it had not been cleaned up in the 5 minutes of the zombie timeout. We think the problem is that the write_vio had nbytes and ndone set to 0. So a WRITE_COMPLETE was sent to the state machine to clean it up. But there was no real write and the watch for client abort handler really needs an EOS to clean up. This change should case an EOS to be sent in this scenario.
This is what the memory utilization looks like after the fix:
ats9.1 with fix
[n7.pdf](https://github.com/apache/trafficserver/files/7017137/n7.pdf)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] duke8253 commented on issue #8265: Memory leaks in Http2Stream
Posted by GitBox <gi...@apache.org>.
duke8253 commented on issue #8265:
URL: https://github.com/apache/trafficserver/issues/8265#issuecomment-902789702
Here's the allocator stats, collected a few minutes apart.
```
Allocated | In-Use | Type Size | Free List Name
--------------------|--------------------|------------|----------------------------------
128188416 | 121688064 | 2048 | memory/hdrStrHeap
127139840 | 121810944 | 2048 | memory/hdrHeap
131072 | 75008 | 256 | memory/httpCacheAltAllocator
44498944 | 44340640 | 3104 | memory/http2StreamAllocator
38819840 | 38809136 | 3568 | memory/http2ClientSessionAllocator
0 | 0 | 144 | memory/RemapPluginsAlloc
16625664 | 16154688 | 1056 | memory/httpServerSessionAllocator
48281600 | 40424800 | 9200 | memory/httpSMAllocator
1081344 | 759264 | 1056 | memory/http1ClientSessionAllocator
0 | 0 | 160 | memory/socksProxyAllocator
45056 | 19776 | 32 | memory/MIMEFieldSDKHandle
0 | 0 | 320 | memory/INKVConnAllocator
2498560 | 2111840 | 160 | memory/INKContAllocator
1327104 | 1125344 | 32 | memory/apiHookAllocator
0 | 0 | 512 | memory/FetchSMAllocator
5373952 | 4499456 | 1024 | memory/ArenaBlock
157810688 | 157628416 | 2048 | memory/hdrStrHeap
161480704 | 161343488 | 2048 | memory/hdrHeap
163840 | 108544 | 256 | memory/httpCacheAltAllocator
51650560 | 51333952 | 3104 | memory/http2StreamAllocator
45670400 | 45242240 | 3568 | memory/http2ClientSessionAllocator
0 | 0 | 144 | memory/RemapPluginsAlloc
19734528 | 19416672 | 1056 | memory/httpServerSessionAllocator
69478400 | 68438800 | 9200 | memory/httpSMAllocator
1081344 | 841632 | 1056 | memory/http1ClientSessionAllocator
0 | 0 | 160 | memory/socksProxyAllocator
45056 | 37056 | 32 | memory/MIMEFieldSDKHandle
0 | 0 | 320 | memory/INKVConnAllocator
3604480 | 3573120 | 160 | memory/INKContAllocator
1916928 | 1904672 | 32 | memory/apiHookAllocator
0 | 0 | 512 | memory/FetchSMAllocator
7733248 | 7616512 | 1024 | memory/ArenaBlock
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] duke8253 closed issue #8265: Memory leaks in Http2Stream
Posted by GitBox <gi...@apache.org>.
duke8253 closed issue #8265:
URL: https://github.com/apache/trafficserver/issues/8265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org