You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Dk Jack <dn...@gmail.com> on 2019/06/02 18:41:34 UTC
Re: memory leak
Does anyone have an idea of how much memory is allocated for processing
each request? In this particular environment (from the graphs), it looks
like none of the memory allocated is being freed. Strange thing is, with no
change to software of configuration (besides a restart of ats), the memory
consumption has gone up over the past couple of week with no appreciable
increase in traffic volumes. It used to leak about 2-3g a day and now it
has gone up to 12-14G a day.
Any idea on where to start looking at and what to look for. I've scoured my
plugin code a few times and it looks clean. The memory dumps show it's
happening in the ATS code, using ATS allocators. Any help is appreciated.
Thanks.
On Fri, May 31, 2019 at 3:42 PM Dk Jack <dn...@gmail.com> wrote:
> stats collected via 'traffic_ctl metric ...' commands...
>
> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0
>
> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dn...@gmail.com> wrote:
>
>> No. I only have healthcheck plugin, stats plugin and my plugin.
>>
>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <sm...@gmail.com>
>> wrote:
>>
>>> Do you have stale while revalidation plug-in? If so, disable.
>>>
>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dn...@gmail.com> wrote:
>>>
>>> > Hi,
>>> > I am running ATS 6.2.1 and I am seeing memory leaks. The link below
>>> shows
>>> > memory dump graphs for a half-hour period (dump freq is 15s). I have a
>>> > custom plugin that's using atscppapi. These graphs are from our
>>> production
>>> > setup where the traffic volume is very high (120M+ req.s/day). We are
>>> > seeing memory growth of 6-8M every minute.
>>> >
>>> > https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0
>>> >
>>> > In my test setup, I don't see the issue. Although, the volume is a lot
>>> less
>>> > in my test setup. I've put debug logs in all places where ATS
>>> allocators
>>> > are showing growth and I see them properly being release in my test
>>> setup.
>>> > Is it possible, ATS is running into some error conditions causing this
>>> > leak? Any pointers on where or how to go about debugging this issue is
>>> > greatly appreciated...
>>> >
>>> > Dk.
>>> >
>>> > PS: I've tried to upgrade to 7.1.6, but I am running into some other
>>> > crashes with my custom plugin enabled.
>>> >
>>>
>>
Re: memory leak
Posted by Dk Jack <dn...@gmail.com>.
btw, I thing I forgot to mention, I am running ATS in a docker container. I
suspect it should not matter, but wanted to provide all the details.
Are there any options that I can enable or commands that I can run to
further diagnose this?
Lief,
We you mentioned disable freelist, did you mean the '-f' option for
traffic_server?
Thanks.
On Sun, Jun 2, 2019 at 5:24 PM Dk Jack <dn...@gmail.com> wrote:
> Thanks Leif for responding,
>
> some questions...
>
> How do I turn off the freelist? Could you please elaborate on your RAM
> disk reference? In my setup, HTTP cache is turned off. Would ram disk still
> come into play?
>
> Is your suggestion 'kill -USR1' different from enabling
> 'proxy.config.dump_mem_info_frequency' config? The graphs I posted where
> made by enabling the dump memory config set to 15s frequency and turning
> the dumped stats into time series data.
>
> Again, thanks for taking the time to respond...
>
> On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>> Did you try turning off the freelist? If you do, you likely want to use
>> jemalloc or tcmalloc instead.
>>
>> If that stops the “leak”, then it’s likely related to how the RAM disk.
>>
>> The other thing to do is to kill -USR1 and look at the allocator usage.
>> Do that a few times, with a few hours between and compare. There’s a script
>> in the tools directory that lets you “diff” two such memory usage dumps.
>>
>> Cheers,
>>
>> — Leif
>>
>> > On Jun 2, 2019, at 12:41, Dk Jack <dn...@gmail.com> wrote:
>> >
>> > Does anyone have an idea of how much memory is allocated for processing
>> > each request? In this particular environment (from the graphs), it looks
>> > like none of the memory allocated is being freed. Strange thing is,
>> with no
>> > change to software of configuration (besides a restart of ats), the
>> memory
>> > consumption has gone up over the past couple of week with no appreciable
>> > increase in traffic volumes. It used to leak about 2-3g a day and now it
>> > has gone up to 12-14G a day.
>> >
>> > Any idea on where to start looking at and what to look for. I've
>> scoured my
>> > plugin code a few times and it looks clean. The memory dumps show it's
>> > happening in the ATS code, using ATS allocators. Any help is
>> appreciated.
>> > Thanks.
>> >
>> >> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dn...@gmail.com> wrote:
>> >>
>> >> stats collected via 'traffic_ctl metric ...' commands...
>> >>
>> >> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0
>> >>
>> >>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dn...@gmail.com> wrote:
>> >>>
>> >>> No. I only have healthcheck plugin, stats plugin and my plugin.
>> >>>
>> >>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <smalenfant@gmail.com
>> >
>> >>> wrote:
>> >>>
>> >>>> Do you have stale while revalidation plug-in? If so, disable.
>> >>>>
>> >>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dn...@gmail.com> wrote:
>> >>>>>
>> >>>>> Hi,
>> >>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link below
>> >>>> shows
>> >>>>> memory dump graphs for a half-hour period (dump freq is 15s). I
>> have a
>> >>>>> custom plugin that's using atscppapi. These graphs are from our
>> >>>> production
>> >>>>> setup where the traffic volume is very high (120M+ req.s/day). We
>> are
>> >>>>> seeing memory growth of 6-8M every minute.
>> >>>>>
>> >>>>> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0
>> >>>>>
>> >>>>> In my test setup, I don't see the issue. Although, the volume is a
>> lot
>> >>>> less
>> >>>>> in my test setup. I've put debug logs in all places where ATS
>> >>>> allocators
>> >>>>> are showing growth and I see them properly being release in my test
>> >>>> setup.
>> >>>>> Is it possible, ATS is running into some error conditions causing
>> this
>> >>>>> leak? Any pointers on where or how to go about debugging this issue
>> is
>> >>>>> greatly appreciated...
>> >>>>>
>> >>>>> Dk.
>> >>>>>
>> >>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some other
>> >>>>> crashes with my custom plugin enabled.
>> >>>>>
>> >>>>
>> >>>
>>
>>
Re: memory leak
Posted by Dk Jack <dn...@gmail.com>.
Opened...
https://github.com/apache/trafficserver/issues/5596
and
https://github.com/apache/trafficserver/issues/5595
On Mon, Jun 3, 2019 at 1:25 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>
> > On Jun 3, 2019, at 2:23 PM, Dk Jack <dn...@gmail.com> wrote:
> >
> > Yeah, I tried moving to 7.1.6. However, I ran into a couple of crashes
> with
> > the new releases…
>
> Those would be good to get reports on, and fixes :) Can you try the latest
> 7.1.1 RC0 that I just posted about?
>
> — leif
>
> >
> > Bhasker.
> >
> > On Mon, Jun 3, 2019 at 12:20 PM Leif Hedstrom <zw...@apache.org> wrote:
> >
> >>
> >>
> >>> On Jun 2, 2019, at 6:24 PM, Dk Jack <dn...@gmail.com> wrote:
> >>>
> >>> Thanks Leif for responding,
> >>>
> >>> some questions...
> >>>
> >>> How do I turn off the freelist? Could you please elaborate on your RAM
> >> disk
> >>> reference? In my setup, HTTP cache is turned off. Would ram disk still
> >> come
> >>> into play?
> >>
> >> Optino to traffic_server:
> >>
> >> -f, --disable_freelist tog false Disable the freelist memory
> >> allocator
> >>
> >>
> >>>
> >>> Is your suggestion 'kill -USR1' different from enabling
> >>> 'proxy.config.dump_mem_info_frequency' config? The graphs I posted
> where
> >>> made by enabling the dump memory config set to 15s frequency and
> turning
> >>> the dumped stats into time series data.
> >>
> >> Yeh, same things, sorry.
> >>
> >> Also, note that ATS v6.2 is no longer supported, really need you to test
> >> it on 7.x or 8.x I think to be able to properly debug.
> >>
> >> — Leif
> >>
> >>>
> >>> Again, thanks for taking the time to respond...
> >>>
> >>> On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org>
> wrote:
> >>>
> >>>> Did you try turning off the freelist? If you do, you likely want to
> use
> >>>> jemalloc or tcmalloc instead.
> >>>>
> >>>> If that stops the “leak”, then it’s likely related to how the RAM
> disk.
> >>>>
> >>>> The other thing to do is to kill -USR1 and look at the allocator
> usage.
> >> Do
> >>>> that a few times, with a few hours between and compare. There’s a
> >> script in
> >>>> the tools directory that lets you “diff” two such memory usage dumps.
> >>>>
> >>>> Cheers,
> >>>>
> >>>> — Leif
> >>>>
> >>>>> On Jun 2, 2019, at 12:41, Dk Jack <dn...@gmail.com> wrote:
> >>>>>
> >>>>> Does anyone have an idea of how much memory is allocated for
> processing
> >>>>> each request? In this particular environment (from the graphs), it
> >> looks
> >>>>> like none of the memory allocated is being freed. Strange thing is,
> >> with
> >>>> no
> >>>>> change to software of configuration (besides a restart of ats), the
> >>>> memory
> >>>>> consumption has gone up over the past couple of week with no
> >> appreciable
> >>>>> increase in traffic volumes. It used to leak about 2-3g a day and now
> >> it
> >>>>> has gone up to 12-14G a day.
> >>>>>
> >>>>> Any idea on where to start looking at and what to look for. I've
> >> scoured
> >>>> my
> >>>>> plugin code a few times and it looks clean. The memory dumps show
> it's
> >>>>> happening in the ATS code, using ATS allocators. Any help is
> >> appreciated.
> >>>>> Thanks.
> >>>>>
> >>>>>> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dn...@gmail.com> wrote:
> >>>>>>
> >>>>>> stats collected via 'traffic_ctl metric ...' commands...
> >>>>>>
> >>>>>> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0
> >>>>>>
> >>>>>>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dn...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> No. I only have healthcheck plugin, stats plugin and my plugin.
> >>>>>>>
> >>>>>>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <
> >> smalenfant@gmail.com>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Do you have stale while revalidation plug-in? If so, disable.
> >>>>>>>>
> >>>>>>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dn...@gmail.com>
> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi,
> >>>>>>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link
> below
> >>>>>>>> shows
> >>>>>>>>> memory dump graphs for a half-hour period (dump freq is 15s). I
> >> have
> >>>> a
> >>>>>>>>> custom plugin that's using atscppapi. These graphs are from our
> >>>>>>>> production
> >>>>>>>>> setup where the traffic volume is very high (120M+ req.s/day). We
> >> are
> >>>>>>>>> seeing memory growth of 6-8M every minute.
> >>>>>>>>>
> >>>>>>>>>
> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0
> >>>>>>>>>
> >>>>>>>>> In my test setup, I don't see the issue. Although, the volume is
> a
> >>>> lot
> >>>>>>>> less
> >>>>>>>>> in my test setup. I've put debug logs in all places where ATS
> >>>>>>>> allocators
> >>>>>>>>> are showing growth and I see them properly being release in my
> test
> >>>>>>>> setup.
> >>>>>>>>> Is it possible, ATS is running into some error conditions causing
> >>>> this
> >>>>>>>>> leak? Any pointers on where or how to go about debugging this
> issue
> >>>> is
> >>>>>>>>> greatly appreciated...
> >>>>>>>>>
> >>>>>>>>> Dk.
> >>>>>>>>>
> >>>>>>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some
> >> other
> >>>>>>>>> crashes with my custom plugin enabled.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
Re: memory leak
Posted by Leif Hedstrom <zw...@apache.org>.
> On Jun 3, 2019, at 2:23 PM, Dk Jack <dn...@gmail.com> wrote:
>
> Yeah, I tried moving to 7.1.6. However, I ran into a couple of crashes with
> the new releases…
Those would be good to get reports on, and fixes :) Can you try the latest 7.1.1 RC0 that I just posted about?
— leif
>
> Bhasker.
>
> On Mon, Jun 3, 2019 at 12:20 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>>
>>
>>> On Jun 2, 2019, at 6:24 PM, Dk Jack <dn...@gmail.com> wrote:
>>>
>>> Thanks Leif for responding,
>>>
>>> some questions...
>>>
>>> How do I turn off the freelist? Could you please elaborate on your RAM
>> disk
>>> reference? In my setup, HTTP cache is turned off. Would ram disk still
>> come
>>> into play?
>>
>> Optino to traffic_server:
>>
>> -f, --disable_freelist tog false Disable the freelist memory
>> allocator
>>
>>
>>>
>>> Is your suggestion 'kill -USR1' different from enabling
>>> 'proxy.config.dump_mem_info_frequency' config? The graphs I posted where
>>> made by enabling the dump memory config set to 15s frequency and turning
>>> the dumped stats into time series data.
>>
>> Yeh, same things, sorry.
>>
>> Also, note that ATS v6.2 is no longer supported, really need you to test
>> it on 7.x or 8.x I think to be able to properly debug.
>>
>> — Leif
>>
>>>
>>> Again, thanks for taking the time to respond...
>>>
>>> On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org> wrote:
>>>
>>>> Did you try turning off the freelist? If you do, you likely want to use
>>>> jemalloc or tcmalloc instead.
>>>>
>>>> If that stops the “leak”, then it’s likely related to how the RAM disk.
>>>>
>>>> The other thing to do is to kill -USR1 and look at the allocator usage.
>> Do
>>>> that a few times, with a few hours between and compare. There’s a
>> script in
>>>> the tools directory that lets you “diff” two such memory usage dumps.
>>>>
>>>> Cheers,
>>>>
>>>> — Leif
>>>>
>>>>> On Jun 2, 2019, at 12:41, Dk Jack <dn...@gmail.com> wrote:
>>>>>
>>>>> Does anyone have an idea of how much memory is allocated for processing
>>>>> each request? In this particular environment (from the graphs), it
>> looks
>>>>> like none of the memory allocated is being freed. Strange thing is,
>> with
>>>> no
>>>>> change to software of configuration (besides a restart of ats), the
>>>> memory
>>>>> consumption has gone up over the past couple of week with no
>> appreciable
>>>>> increase in traffic volumes. It used to leak about 2-3g a day and now
>> it
>>>>> has gone up to 12-14G a day.
>>>>>
>>>>> Any idea on where to start looking at and what to look for. I've
>> scoured
>>>> my
>>>>> plugin code a few times and it looks clean. The memory dumps show it's
>>>>> happening in the ATS code, using ATS allocators. Any help is
>> appreciated.
>>>>> Thanks.
>>>>>
>>>>>> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dn...@gmail.com> wrote:
>>>>>>
>>>>>> stats collected via 'traffic_ctl metric ...' commands...
>>>>>>
>>>>>> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0
>>>>>>
>>>>>>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dn...@gmail.com> wrote:
>>>>>>>
>>>>>>> No. I only have healthcheck plugin, stats plugin and my plugin.
>>>>>>>
>>>>>>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <
>> smalenfant@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Do you have stale while revalidation plug-in? If so, disable.
>>>>>>>>
>>>>>>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dn...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link below
>>>>>>>> shows
>>>>>>>>> memory dump graphs for a half-hour period (dump freq is 15s). I
>> have
>>>> a
>>>>>>>>> custom plugin that's using atscppapi. These graphs are from our
>>>>>>>> production
>>>>>>>>> setup where the traffic volume is very high (120M+ req.s/day). We
>> are
>>>>>>>>> seeing memory growth of 6-8M every minute.
>>>>>>>>>
>>>>>>>>> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0
>>>>>>>>>
>>>>>>>>> In my test setup, I don't see the issue. Although, the volume is a
>>>> lot
>>>>>>>> less
>>>>>>>>> in my test setup. I've put debug logs in all places where ATS
>>>>>>>> allocators
>>>>>>>>> are showing growth and I see them properly being release in my test
>>>>>>>> setup.
>>>>>>>>> Is it possible, ATS is running into some error conditions causing
>>>> this
>>>>>>>>> leak? Any pointers on where or how to go about debugging this issue
>>>> is
>>>>>>>>> greatly appreciated...
>>>>>>>>>
>>>>>>>>> Dk.
>>>>>>>>>
>>>>>>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some
>> other
>>>>>>>>> crashes with my custom plugin enabled.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>>>
>>
>>
Re: memory leak
Posted by Dk Jack <dn...@gmail.com>.
Yeah, I tried moving to 7.1.6. However, I ran into a couple of crashes with
the new releases...
Bhasker.
On Mon, Jun 3, 2019 at 12:20 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>
> > On Jun 2, 2019, at 6:24 PM, Dk Jack <dn...@gmail.com> wrote:
> >
> > Thanks Leif for responding,
> >
> > some questions...
> >
> > How do I turn off the freelist? Could you please elaborate on your RAM
> disk
> > reference? In my setup, HTTP cache is turned off. Would ram disk still
> come
> > into play?
>
> Optino to traffic_server:
>
> -f, --disable_freelist tog false Disable the freelist memory
> allocator
>
>
> >
> > Is your suggestion 'kill -USR1' different from enabling
> > 'proxy.config.dump_mem_info_frequency' config? The graphs I posted where
> > made by enabling the dump memory config set to 15s frequency and turning
> > the dumped stats into time series data.
>
> Yeh, same things, sorry.
>
> Also, note that ATS v6.2 is no longer supported, really need you to test
> it on 7.x or 8.x I think to be able to properly debug.
>
> — Leif
>
> >
> > Again, thanks for taking the time to respond...
> >
> > On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org> wrote:
> >
> >> Did you try turning off the freelist? If you do, you likely want to use
> >> jemalloc or tcmalloc instead.
> >>
> >> If that stops the “leak”, then it’s likely related to how the RAM disk.
> >>
> >> The other thing to do is to kill -USR1 and look at the allocator usage.
> Do
> >> that a few times, with a few hours between and compare. There’s a
> script in
> >> the tools directory that lets you “diff” two such memory usage dumps.
> >>
> >> Cheers,
> >>
> >> — Leif
> >>
> >>> On Jun 2, 2019, at 12:41, Dk Jack <dn...@gmail.com> wrote:
> >>>
> >>> Does anyone have an idea of how much memory is allocated for processing
> >>> each request? In this particular environment (from the graphs), it
> looks
> >>> like none of the memory allocated is being freed. Strange thing is,
> with
> >> no
> >>> change to software of configuration (besides a restart of ats), the
> >> memory
> >>> consumption has gone up over the past couple of week with no
> appreciable
> >>> increase in traffic volumes. It used to leak about 2-3g a day and now
> it
> >>> has gone up to 12-14G a day.
> >>>
> >>> Any idea on where to start looking at and what to look for. I've
> scoured
> >> my
> >>> plugin code a few times and it looks clean. The memory dumps show it's
> >>> happening in the ATS code, using ATS allocators. Any help is
> appreciated.
> >>> Thanks.
> >>>
> >>>> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dn...@gmail.com> wrote:
> >>>>
> >>>> stats collected via 'traffic_ctl metric ...' commands...
> >>>>
> >>>> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0
> >>>>
> >>>>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dn...@gmail.com> wrote:
> >>>>>
> >>>>> No. I only have healthcheck plugin, stats plugin and my plugin.
> >>>>>
> >>>>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <
> smalenfant@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Do you have stale while revalidation plug-in? If so, disable.
> >>>>>>
> >>>>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dn...@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Hi,
> >>>>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link below
> >>>>>> shows
> >>>>>>> memory dump graphs for a half-hour period (dump freq is 15s). I
> have
> >> a
> >>>>>>> custom plugin that's using atscppapi. These graphs are from our
> >>>>>> production
> >>>>>>> setup where the traffic volume is very high (120M+ req.s/day). We
> are
> >>>>>>> seeing memory growth of 6-8M every minute.
> >>>>>>>
> >>>>>>> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0
> >>>>>>>
> >>>>>>> In my test setup, I don't see the issue. Although, the volume is a
> >> lot
> >>>>>> less
> >>>>>>> in my test setup. I've put debug logs in all places where ATS
> >>>>>> allocators
> >>>>>>> are showing growth and I see them properly being release in my test
> >>>>>> setup.
> >>>>>>> Is it possible, ATS is running into some error conditions causing
> >> this
> >>>>>>> leak? Any pointers on where or how to go about debugging this issue
> >> is
> >>>>>>> greatly appreciated...
> >>>>>>>
> >>>>>>> Dk.
> >>>>>>>
> >>>>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some
> other
> >>>>>>> crashes with my custom plugin enabled.
> >>>>>>>
> >>>>>>
> >>>>>
> >>
> >>
>
>
Re: memory leak
Posted by Leif Hedstrom <zw...@apache.org>.
> On Jun 2, 2019, at 6:24 PM, Dk Jack <dn...@gmail.com> wrote:
>
> Thanks Leif for responding,
>
> some questions...
>
> How do I turn off the freelist? Could you please elaborate on your RAM disk
> reference? In my setup, HTTP cache is turned off. Would ram disk still come
> into play?
Optino to traffic_server:
-f, --disable_freelist tog false Disable the freelist memory allocator
>
> Is your suggestion 'kill -USR1' different from enabling
> 'proxy.config.dump_mem_info_frequency' config? The graphs I posted where
> made by enabling the dump memory config set to 15s frequency and turning
> the dumped stats into time series data.
Yeh, same things, sorry.
Also, note that ATS v6.2 is no longer supported, really need you to test it on 7.x or 8.x I think to be able to properly debug.
— Leif
>
> Again, thanks for taking the time to respond...
>
> On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>> Did you try turning off the freelist? If you do, you likely want to use
>> jemalloc or tcmalloc instead.
>>
>> If that stops the “leak”, then it’s likely related to how the RAM disk.
>>
>> The other thing to do is to kill -USR1 and look at the allocator usage. Do
>> that a few times, with a few hours between and compare. There’s a script in
>> the tools directory that lets you “diff” two such memory usage dumps.
>>
>> Cheers,
>>
>> — Leif
>>
>>> On Jun 2, 2019, at 12:41, Dk Jack <dn...@gmail.com> wrote:
>>>
>>> Does anyone have an idea of how much memory is allocated for processing
>>> each request? In this particular environment (from the graphs), it looks
>>> like none of the memory allocated is being freed. Strange thing is, with
>> no
>>> change to software of configuration (besides a restart of ats), the
>> memory
>>> consumption has gone up over the past couple of week with no appreciable
>>> increase in traffic volumes. It used to leak about 2-3g a day and now it
>>> has gone up to 12-14G a day.
>>>
>>> Any idea on where to start looking at and what to look for. I've scoured
>> my
>>> plugin code a few times and it looks clean. The memory dumps show it's
>>> happening in the ATS code, using ATS allocators. Any help is appreciated.
>>> Thanks.
>>>
>>>> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dn...@gmail.com> wrote:
>>>>
>>>> stats collected via 'traffic_ctl metric ...' commands...
>>>>
>>>> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0
>>>>
>>>>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dn...@gmail.com> wrote:
>>>>>
>>>>> No. I only have healthcheck plugin, stats plugin and my plugin.
>>>>>
>>>>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <sm...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Do you have stale while revalidation plug-in? If so, disable.
>>>>>>
>>>>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dn...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link below
>>>>>> shows
>>>>>>> memory dump graphs for a half-hour period (dump freq is 15s). I have
>> a
>>>>>>> custom plugin that's using atscppapi. These graphs are from our
>>>>>> production
>>>>>>> setup where the traffic volume is very high (120M+ req.s/day). We are
>>>>>>> seeing memory growth of 6-8M every minute.
>>>>>>>
>>>>>>> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0
>>>>>>>
>>>>>>> In my test setup, I don't see the issue. Although, the volume is a
>> lot
>>>>>> less
>>>>>>> in my test setup. I've put debug logs in all places where ATS
>>>>>> allocators
>>>>>>> are showing growth and I see them properly being release in my test
>>>>>> setup.
>>>>>>> Is it possible, ATS is running into some error conditions causing
>> this
>>>>>>> leak? Any pointers on where or how to go about debugging this issue
>> is
>>>>>>> greatly appreciated...
>>>>>>>
>>>>>>> Dk.
>>>>>>>
>>>>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some other
>>>>>>> crashes with my custom plugin enabled.
>>>>>>>
>>>>>>
>>>>>
>>
>>
Re: memory leak
Posted by Dk Jack <dn...@gmail.com>.
Thanks Leif for responding,
some questions...
How do I turn off the freelist? Could you please elaborate on your RAM disk
reference? In my setup, HTTP cache is turned off. Would ram disk still come
into play?
Is your suggestion 'kill -USR1' different from enabling
'proxy.config.dump_mem_info_frequency' config? The graphs I posted where
made by enabling the dump memory config set to 15s frequency and turning
the dumped stats into time series data.
Again, thanks for taking the time to respond...
On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org> wrote:
> Did you try turning off the freelist? If you do, you likely want to use
> jemalloc or tcmalloc instead.
>
> If that stops the “leak”, then it’s likely related to how the RAM disk.
>
> The other thing to do is to kill -USR1 and look at the allocator usage. Do
> that a few times, with a few hours between and compare. There’s a script in
> the tools directory that lets you “diff” two such memory usage dumps.
>
> Cheers,
>
> — Leif
>
> > On Jun 2, 2019, at 12:41, Dk Jack <dn...@gmail.com> wrote:
> >
> > Does anyone have an idea of how much memory is allocated for processing
> > each request? In this particular environment (from the graphs), it looks
> > like none of the memory allocated is being freed. Strange thing is, with
> no
> > change to software of configuration (besides a restart of ats), the
> memory
> > consumption has gone up over the past couple of week with no appreciable
> > increase in traffic volumes. It used to leak about 2-3g a day and now it
> > has gone up to 12-14G a day.
> >
> > Any idea on where to start looking at and what to look for. I've scoured
> my
> > plugin code a few times and it looks clean. The memory dumps show it's
> > happening in the ATS code, using ATS allocators. Any help is appreciated.
> > Thanks.
> >
> >> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dn...@gmail.com> wrote:
> >>
> >> stats collected via 'traffic_ctl metric ...' commands...
> >>
> >> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0
> >>
> >>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dn...@gmail.com> wrote:
> >>>
> >>> No. I only have healthcheck plugin, stats plugin and my plugin.
> >>>
> >>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <sm...@gmail.com>
> >>> wrote:
> >>>
> >>>> Do you have stale while revalidation plug-in? If so, disable.
> >>>>
> >>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dn...@gmail.com> wrote:
> >>>>>
> >>>>> Hi,
> >>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link below
> >>>> shows
> >>>>> memory dump graphs for a half-hour period (dump freq is 15s). I have
> a
> >>>>> custom plugin that's using atscppapi. These graphs are from our
> >>>> production
> >>>>> setup where the traffic volume is very high (120M+ req.s/day). We are
> >>>>> seeing memory growth of 6-8M every minute.
> >>>>>
> >>>>> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0
> >>>>>
> >>>>> In my test setup, I don't see the issue. Although, the volume is a
> lot
> >>>> less
> >>>>> in my test setup. I've put debug logs in all places where ATS
> >>>> allocators
> >>>>> are showing growth and I see them properly being release in my test
> >>>> setup.
> >>>>> Is it possible, ATS is running into some error conditions causing
> this
> >>>>> leak? Any pointers on where or how to go about debugging this issue
> is
> >>>>> greatly appreciated...
> >>>>>
> >>>>> Dk.
> >>>>>
> >>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some other
> >>>>> crashes with my custom plugin enabled.
> >>>>>
> >>>>
> >>>
>
>
Re: memory leak
Posted by Leif Hedstrom <zw...@apache.org>.
Did you try turning off the freelist? If you do, you likely want to use jemalloc or tcmalloc instead.
If that stops the “leak”, then it’s likely related to how the RAM disk.
The other thing to do is to kill -USR1 and look at the allocator usage. Do that a few times, with a few hours between and compare. There’s a script in the tools directory that lets you “diff” two such memory usage dumps.
Cheers,
— Leif
> On Jun 2, 2019, at 12:41, Dk Jack <dn...@gmail.com> wrote:
>
> Does anyone have an idea of how much memory is allocated for processing
> each request? In this particular environment (from the graphs), it looks
> like none of the memory allocated is being freed. Strange thing is, with no
> change to software of configuration (besides a restart of ats), the memory
> consumption has gone up over the past couple of week with no appreciable
> increase in traffic volumes. It used to leak about 2-3g a day and now it
> has gone up to 12-14G a day.
>
> Any idea on where to start looking at and what to look for. I've scoured my
> plugin code a few times and it looks clean. The memory dumps show it's
> happening in the ATS code, using ATS allocators. Any help is appreciated.
> Thanks.
>
>> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dn...@gmail.com> wrote:
>>
>> stats collected via 'traffic_ctl metric ...' commands...
>>
>> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0
>>
>>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dn...@gmail.com> wrote:
>>>
>>> No. I only have healthcheck plugin, stats plugin and my plugin.
>>>
>>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <sm...@gmail.com>
>>> wrote:
>>>
>>>> Do you have stale while revalidation plug-in? If so, disable.
>>>>
>>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dn...@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link below
>>>> shows
>>>>> memory dump graphs for a half-hour period (dump freq is 15s). I have a
>>>>> custom plugin that's using atscppapi. These graphs are from our
>>>> production
>>>>> setup where the traffic volume is very high (120M+ req.s/day). We are
>>>>> seeing memory growth of 6-8M every minute.
>>>>>
>>>>> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0
>>>>>
>>>>> In my test setup, I don't see the issue. Although, the volume is a lot
>>>> less
>>>>> in my test setup. I've put debug logs in all places where ATS
>>>> allocators
>>>>> are showing growth and I see them properly being release in my test
>>>> setup.
>>>>> Is it possible, ATS is running into some error conditions causing this
>>>>> leak? Any pointers on where or how to go about debugging this issue is
>>>>> greatly appreciated...
>>>>>
>>>>> Dk.
>>>>>
>>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some other
>>>>> crashes with my custom plugin enabled.
>>>>>
>>>>
>>>