You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Steve Malenfant <sm...@gmail.com> on 2018/10/17 19:00:39 UTC
Re: Memory usage - Virtual Machine - 7.1.x
I tried the stats_over_http plugin today and it's behaving the same.
memory/hdrHeap and memory/hdrStrHeap are increasing on every requests.
Will file an issue. Not only a Virtual Machine issue.
Tried to get jemalloc to help us but I believe we failed. No output.
How to reproduce?
#!/bin/bash
while true; do
curl http://<ip>/_stats >/dev/null
done
Steve
On Mon, Sep 10, 2018 at 7:21 AM Steve Malenfant <sm...@gmail.com>
wrote:
> Ever since we upgraded to 6.x or 7.x, it seems like we have issues with
> memory utilization climbing over time in a 4GB VM. It takes in average
> about 5 days before it starts swapping.
>
> Current version: 7.1.4
>
> There is only a few requests getting to those VMs and they are creating
> quite a problem on the VM hosts due to excessive swapping (IO). Most of
> those requests, if not all are using the astats_over_http plugin for
> Traffic Control.
>
> I've condensed the memory dump just to show what's being used.
> Seems like hdrStrHeap seems to be using quite a bit of memory and keeps
> increasing over time.
>
> Is there anything specific I would need to look into?
>
>
> -----------------------------------------------------------------------------------------
> Allocated | In-Use | Type Size | Free List Name
>
> --------------------|--------------------|------------|----------------------------------
> 2097152 | 0 | 32768 |
> memory/ioBufAllocator[8]
> 524288 | 0 | 8192 |
> memory/ioBufAllocator[6]
> 96993280 | 96661504 | 4096 |
> memory/ioBufAllocator[5]
> 16384 | 128 | 128 |
> memory/ioBufAllocator[0]
> 73728 | 45312 | 96 |
> memory/eventAllocator
> 1407600 | 1399440 | 80 |
> memory/mutexAllocator
> 24576 | 22016 | 64 |
> memory/ioBlockAllocator
> 1150560 | 1145328 | 48 |
> memory/ioDataAllocator
> 65280 | 62400 | 240 | memory/ioAllocator
> 97760 | 25568 | 752 |
> memory/netVCAllocator
> 114400 | 0 | 880 |
> memory/sslNetVCAllocator
> 67712 | 33856 | 33856 |
> memory/dnsBufAllocator
> 163840 | 0 | 1280 |
> memory/dnsEntryAllocator
> 4096 | 16 | 16 |
> memory/expiryQueueEntry
> 8192 | 64 | 64 |
> memory/refCountCacheHashingValueAllocator
> 296960 | 2320 | 2320 |
> memory/hostDBContAllocator
> 1162870784 | 1162797056 | 2048 | memory/hdrStrHeap
> 1162870784 | 1162862592 | 2048 | memory/hdrHeap
> 24576 | 6144 | 192 |
> memory/httpServerSessionAllocator
> 1990656 | 0 | 7776 |
> memory/httpSMAllocator
> 114688 | 30464 | 896 |
> memory/http1ClientSessionAllocator
> 24576 | 6624 | 96 |
> memory/INKContAllocator
> 4096 | 224 | 32 |
> memory/apiHookAllocator
> 262144 | 0 | 1024 | memory/ArenaBlock
> 2431268112 | 2425101056 | | TOTAL
>
> -----------------------------------------------------------------------------------------
>
> Steve
>
Re: Memory usage - Virtual Machine - 7.1.x
Posted by Leif Hedstrom <zw...@apache.org>.
> On Oct 31, 2018, at 10:00 AM, Alan Carroll <so...@oath.com> wrote:
>
> Looking at the memory dump, my first guess would be you have a lot of stalled transactions that never got cleaned up. This is based on the ioBufAllocator[5], which IIRC is the default size for the initial read. The hdrStrHeap and hdrHeap are used for storing request / response headers in memory. Those being so large seems to indicate that data isn't getting cleaned up.
Oh, we should have followed up on this. We worked with Steve for a bit, and tracked it down to the stale-while-revalidate plugin. Since this plugin is dead now, I don’t think it’s worthwhile to try to fix it (but, we’ll take patches for 7.1.x if anyone wants to work on it).
We need a better alternative for stale-while-revalidate, since the plugin has always been pretty darn crippled :).
Cheers,
— leif
Re: Memory usage - Virtual Machine - 7.1.x
Posted by Alan Carroll <so...@oath.com>.
Looking at the memory dump, my first guess would be you have a lot of
stalled transactions that never got cleaned up. This is based on the
ioBufAllocator[5], which IIRC is the default size for the initial read. The
hdrStrHeap and hdrHeap are used for storing request / response headers in
memory. Those being so large seems to indicate that data isn't getting
cleaned up.
On Wed, Oct 17, 2018 at 2:00 PM Steve Malenfant <sm...@gmail.com>
wrote:
> I tried the stats_over_http plugin today and it's behaving the same.
> memory/hdrHeap and memory/hdrStrHeap are increasing on every requests.
>
> Will file an issue. Not only a Virtual Machine issue.
>
> Tried to get jemalloc to help us but I believe we failed. No output.
>
> How to reproduce?
>
> #!/bin/bash
>
> while true; do
>
> curl http://<ip>/_stats >/dev/null
>
> done
>
> Steve
>
> On Mon, Sep 10, 2018 at 7:21 AM Steve Malenfant <sm...@gmail.com>
> wrote:
>
>> Ever since we upgraded to 6.x or 7.x, it seems like we have issues with
>> memory utilization climbing over time in a 4GB VM. It takes in average
>> about 5 days before it starts swapping.
>>
>> Current version: 7.1.4
>>
>> There is only a few requests getting to those VMs and they are creating
>> quite a problem on the VM hosts due to excessive swapping (IO). Most of
>> those requests, if not all are using the astats_over_http plugin for
>> Traffic Control.
>>
>> I've condensed the memory dump just to show what's being used.
>> Seems like hdrStrHeap seems to be using quite a bit of memory and keeps
>> increasing over time.
>>
>> Is there anything specific I would need to look into?
>>
>>
>> -----------------------------------------------------------------------------------------
>> Allocated | In-Use | Type Size | Free List Name
>>
>> --------------------|--------------------|------------|----------------------------------
>> 2097152 | 0 | 32768 |
>> memory/ioBufAllocator[8]
>> 524288 | 0 | 8192 |
>> memory/ioBufAllocator[6]
>> 96993280 | 96661504 | 4096 |
>> memory/ioBufAllocator[5]
>> 16384 | 128 | 128 |
>> memory/ioBufAllocator[0]
>> 73728 | 45312 | 96 |
>> memory/eventAllocator
>> 1407600 | 1399440 | 80 |
>> memory/mutexAllocator
>> 24576 | 22016 | 64 |
>> memory/ioBlockAllocator
>> 1150560 | 1145328 | 48 |
>> memory/ioDataAllocator
>> 65280 | 62400 | 240 | memory/ioAllocator
>> 97760 | 25568 | 752 |
>> memory/netVCAllocator
>> 114400 | 0 | 880 |
>> memory/sslNetVCAllocator
>> 67712 | 33856 | 33856 |
>> memory/dnsBufAllocator
>> 163840 | 0 | 1280 |
>> memory/dnsEntryAllocator
>> 4096 | 16 | 16 |
>> memory/expiryQueueEntry
>> 8192 | 64 | 64 |
>> memory/refCountCacheHashingValueAllocator
>> 296960 | 2320 | 2320 |
>> memory/hostDBContAllocator
>> 1162870784 | 1162797056 | 2048 | memory/hdrStrHeap
>> 1162870784 | 1162862592 | 2048 | memory/hdrHeap
>> 24576 | 6144 | 192 |
>> memory/httpServerSessionAllocator
>> 1990656 | 0 | 7776 |
>> memory/httpSMAllocator
>> 114688 | 30464 | 896 |
>> memory/http1ClientSessionAllocator
>> 24576 | 6624 | 96 |
>> memory/INKContAllocator
>> 4096 | 224 | 32 |
>> memory/apiHookAllocator
>> 262144 | 0 | 1024 | memory/ArenaBlock
>> 2431268112 | 2425101056 | | TOTAL
>>
>> -----------------------------------------------------------------------------------------
>>
>> Steve
>>
>
--
*Beware the fisherman who's casting out his line in to a dried up riverbed.*
*Oh don't try to tell him 'cause he won't believe. Throw some bread to the
ducks instead.*
*It's easier that way. *- Genesis : Duke : VI 25-28