Re: ATS Memory Usage Steady Increase

[Hongfei Zhang <ho...@gmail.com>]

Thanks Fei.  There weren’t SSL/TLS sessions in our environment but I do feel some of memory are being held by ‘dormant’ sessions. The total of amount of memory held by freelist (44G) was however surprisingly high. Majority of that (99%) are allocated through and held by ioBufAllocator. I am wondering if there is anyway to limit the size of these freelists, also curious what caused the ‘Allocated’ to continue to go up and why the ‘In-Use’ did not go to zero after user traffic stops (and all of the keep-alive session times out). 

I am also puzzled by the line memory/RamCacheLRUEntry shows only 5.2M, where traffic_top shows about 32GB ram cache used. 


Thanks,
-Hongfei

> On Dec 17, 2020, at 3:32 PM, Fei Deng <du...@apache.org> wrote:
> 
> Not saying this is the exact cause, but we've seen similar behavior
> previously. The reason for our issue was the session cache size was set to
> a size too big compared to the ram size, and since the sessions stored in
> the cache are only removed when the cache is full, and inserting new
> sessions caused it to trigger *removeOldestSession*. You might want to
> check your configurations related to this feature
> *proxy.config.ssl.session_cache.size*.
> 
> On Thu, Dec 17, 2020 at 1:52 PM Hongfei Zhang <ho...@gmail.com> wrote:
> 
>> Hi Folks,
>> 
>> Based on information provided
>> https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/performance/index.en.html#memory-allocation and
>> with a fixed ram_cache.size setting (32GB), we expected the memory usage to
>> be plateaued a couple of days usage.    This is not however what we saw in
>> multiple production environments. It seemed the memory usage increases
>> steadily overtime, abeilt as a slow pace once the system’s memory usage
>> reaches 80-85% (there aren’t many other processes running on the system),
>> until to a point ATS process is killed by kernel (oom kill) or human
>> intervention (server restart). On a system with 192GB ram (32GB used for
>> RAM disk, and ATS configured to use up to 32GB ram cache), peaking
>> streaming throughput at 10Gbps, ATS has to be killed/restared in about 2
>> weeks.  At peak hours, there are about 5k-6k client connections and less
>> than 1k upstream connections (to mid tier caches).
>> 
>> We did some analysis on the Freelist dump (kill -USR1 pid) output (an
>> example is attached) and found the allocated in ioBufAllocator[0-14] slots
>> appeared to be main contributor to the total and also likely to be the
>> source of the increase overtime.
>> 
>> In terms of configurations and plugin usage,  in addition to ram_cache
>> setting to 32GB, we also
>> changed proxy.config.http.default_buffer_water_mark INT 15000000 (from
>> default 64k) to allow the entire video segment to be buffered on the
>> upstream connection to avoid client starvation issue when the first client
>> comes from a slow draining link, and
>> proxy.config.cache.target_fragment_size INT 4096 to allow upstream chunked
>> responses to be written into cache storage timely.  There is no connection
>> limits (# of connections appeared to be always in the normal range).  The
>> inactivity timeout values are fairly low (<120 secs).The only plugin we
>> used is header_rewrite.so. No https, no http/2.
>> 
>> I would appreciate if someone can shed some lights on how to further track
>> this down, and any practical tips for short term mitigation. In particular:
>> 1. Inside HttpSM, which states require allocate/re-use ioBuf? Is there a
>> way to put a ceiling on each slot or total allocation?
>> 2. Is the ioBufAllocation ceiling a function of total connections in which
>> case I should set a connection limit?
>> 3. The memory/RamCacheLRUEntry shows 5.2M, how is this related to the
>> actual ram_cache usage reported by traffic_top (32GB used)?
>> 4. At the point of the freelist dump, ATS process size was 78GB, the
>> freelist total showed about 44GB, with 32GB ram_cache used (traffic_top
>> reports). Assuming these two number are not overlapping, I also know the
>> in-memory (disk) directory entry cache takes at least 10GB, then these
>> numbers do add up. 44+32+10 >> 78. What am I missing?
>> 
>> 
>> Thanks,
>> -Hongfei
>> 

Re: ATS Memory Usage Steady Increase

[Masaori Koshiba <ma...@apache.org>]

Hi Hongfei,

Recently, we faced a memory leak on the following redirect with 8.1.1[*1].
The fix[*2] is coming in the next release, 8.1.2 and 9.0.1.
If you didn't have your leak with 8.1.0, it might be the same leak.

> 1. Inside HttpSM, which states require allocate/re-use ioBuf? Is there a
way to put a ceiling on each slot or total allocation?
IOBuffer(s) are allocated in many states. In general, Address Sanitizer
(ASan) is helpful to detect memory leaks.

1. Build ATS with `--enable-asan` option
2. Run Traffic Server without freelist (--fF)

[*1] https://github.com/apache/trafficserver/issues/7380
[*2] https://github.com/apache/trafficserver/pull/7401

Thanks,
Masaori

On Tue, Jan 12, 2021 at 4:19 AM Hongfei Zhang <ho...@gmail.com> wrote:

> Thanks Fei.  There weren’t SSL/TLS sessions in our environment but I do
> feel some of memory are being held by ‘dormant’ sessions. The total of
> amount of memory held by freelist (44G) was however surprisingly high.
> Majority of that (99%) are allocated through and held by ioBufAllocator. I
> am wondering if there is anyway to limit the size of these freelists, also
> curious what caused the ‘Allocated’ to continue to go up and why the
> ‘In-Use’ did not go to zero after user traffic stops (and all of the
> keep-alive session times out).
>
> I am also puzzled by the line memory/RamCacheLRUEntry shows only 5.2M,
> where traffic_top shows about 32GB ram cache used.
>
>
> Thanks,
> -Hongfei
>
> > On Dec 17, 2020, at 3:32 PM, Fei Deng <du...@apache.org> wrote:
> >
> > Not saying this is the exact cause, but we've seen similar behavior
> > previously. The reason for our issue was the session cache size was set
> to
> > a size too big compared to the ram size, and since the sessions stored in
> > the cache are only removed when the cache is full, and inserting new
> > sessions caused it to trigger *removeOldestSession*. You might want to
> > check your configurations related to this feature
> > *proxy.config.ssl.session_cache.size*.
> >
> > On Thu, Dec 17, 2020 at 1:52 PM Hongfei Zhang <ho...@gmail.com>
> wrote:
> >
> >> Hi Folks,
> >>
> >> Based on information provided
> >>
> https://docs.trafficserver.apache.org/en/8.1.x/admin-guide/performance/index.en.html#memory-allocation
> and
> >> with a fixed ram_cache.size setting (32GB), we expected the memory
> usage to
> >> be plateaued a couple of days usage.    This is not however what we saw
> in
> >> multiple production environments. It seemed the memory usage increases
> >> steadily overtime, abeilt as a slow pace once the system’s memory usage
> >> reaches 80-85% (there aren’t many other processes running on the
> system),
> >> until to a point ATS process is killed by kernel (oom kill) or human
> >> intervention (server restart). On a system with 192GB ram (32GB used for
> >> RAM disk, and ATS configured to use up to 32GB ram cache), peaking
> >> streaming throughput at 10Gbps, ATS has to be killed/restared in about 2
> >> weeks.  At peak hours, there are about 5k-6k client connections and less
> >> than 1k upstream connections (to mid tier caches).
> >>
> >> We did some analysis on the Freelist dump (kill -USR1 pid) output (an
> >> example is attached) and found the allocated in ioBufAllocator[0-14]
> slots
> >> appeared to be main contributor to the total and also likely to be the
> >> source of the increase overtime.
> >>
> >> In terms of configurations and plugin usage,  in addition to ram_cache
> >> setting to 32GB, we also
> >> changed proxy.config.http.default_buffer_water_mark INT 15000000 (from
> >> default 64k) to allow the entire video segment to be buffered on the
> >> upstream connection to avoid client starvation issue when the first
> client
> >> comes from a slow draining link, and
> >> proxy.config.cache.target_fragment_size INT 4096 to allow upstream
> chunked
> >> responses to be written into cache storage timely.  There is no
> connection
> >> limits (# of connections appeared to be always in the normal range).
> The
> >> inactivity timeout values are fairly low (<120 secs).The only plugin we
> >> used is header_rewrite.so. No https, no http/2.
> >>
> >> I would appreciate if someone can shed some lights on how to further
> track
> >> this down, and any practical tips for short term mitigation. In
> particular:
> >> 1. Inside HttpSM, which states require allocate/re-use ioBuf? Is there a
> >> way to put a ceiling on each slot or total allocation?
> >> 2. Is the ioBufAllocation ceiling a function of total connections in
> which
> >> case I should set a connection limit?
> >> 3. The memory/RamCacheLRUEntry shows 5.2M, how is this related to the
> >> actual ram_cache usage reported by traffic_top (32GB used)?
> >> 4. At the point of the freelist dump, ATS process size was 78GB, the
> >> freelist total showed about 44GB, with 32GB ram_cache used (traffic_top
> >> reports). Assuming these two number are not overlapping, I also know the
> >> in-memory (disk) directory entry cache takes at least 10GB, then these
> >> numbers do add up. 44+32+10 >> 78. What am I missing?
> >>
> >>
> >> Thanks,
> >> -Hongfei
> >>
>
>