You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by "ming.zym@gmail.com" <mi...@gmail.com> on 2012/12/08 17:47:51 UTC
memory management enhancement in TS-1006
laster week, we have submited the patches for TS-1006,
https://issues.apache.org/jira/browse/TS-1006 .
the memory issue is the most critical issue here as we are running TS in
heavy duty, here is the 2nd version of our memory patch in TS-1006:
https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
if you have some issue with memory problem, please try it and feed back.
and if you are interesting in the detail and here is the docs for the
detail, but in Chinese:
https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
these patches done by Zhang Yunkai, guided by Chen Bin. the first
version contributed by Chen Bin(kuotai), and not intend to commit.
there is another memory issue in the ram_cache, where we'd like to fix
it if we have TS-1006 accepted, maybe we can put our interim solution
later.
thanks
--
Zhao Yongming
aka: zym @ apache.org, yonghao @ taobao.com
Re: memory management enhancement in TS-1006
Posted by Yunkai Zhang <yu...@gmail.com>.
在 2012年12月10日星期一，Leif Hedstrom 写道：
> On 12/8/12 9:47 AM, ming.zym@gmail.com wrote:
>
>> laster week, we have submited the patches for TS-1006,
>> https://issues.apache.org/**jira/browse/TS-1006<https://issues.apache.org/jira/browse/TS-1006>.
>>
>
> Ok, I'm reading TS-1006 again, and i'll take a look at the patches. But
> just to be sure, the idea here is to fix something that's an apparently
> "leak" or wasteful allocation scheme related to the RAM cache ?
Hi Leif:
These patches just try to limited the total memory size allocated
by InkFreeList Allocator which will be used both by Class Object and RAM
cache(not only for RAM cache, and they doesn't modify the code about
RAM cache).
We have another patch developed by Bin Chen which fasten the memory size
only used by RAM cache, but we havn't submitted it to here now, I'm going
to trim it, I need some time to understand it, and maybe do some
optimization or clear work for it before we send it to here.
>
> -- Leif
>
>
--
Yunkai Zhang
Work at Taobao
Re: memory management enhancement in TS-1006
Posted by Yunkai Zhang <yu...@gmail.com>.
在 2012年12月10日星期一，Leif Hedstrom 写道：
> On 12/8/12 9:47 AM, ming.zym@gmail.com wrote:
>
>> laster week, we have submited the patches for TS-1006,
>> https://issues.apache.org/**jira/browse/TS-1006<https://issues.apache.org/jira/browse/TS-1006>.
>>
>
> Ok, I'm reading TS-1006 again, and i'll take a look at the patches. But
> just to be sure, the idea here is to fix something that's an apparently
> "leak" or wasteful allocation scheme related to the RAM cache ?
Hi Leif:
These patches just try to limited the total memory size allocated
by InkFreeList Allocator which will be used both by Class Object and RAM
cache(not only for RAM cache, and they doesn't modify the code about
RAM cache).
We have another patch developed by Bin Chen which fasten the memory size
only used by RAM cache, but we havn't submitted it to here now, I'm going
to trim it, I need some time to understand it, and maybe do some
optimization or clear work for it before we send it to here.
>
> -- Leif
>
>
--
Yunkai Zhang
Work at Taobao
Re: memory management enhancement in TS-1006
Posted by Leif Hedstrom <zw...@apache.org>.
On 12/8/12 9:47 AM, ming.zym@gmail.com wrote:
> laster week, we have submited the patches for TS-1006,
> https://issues.apache.org/jira/browse/TS-1006 .
Ok, I'm reading TS-1006 again, and i'll take a look at the patches. But just
to be sure, the idea here is to fix something that's an apparently "leak" or
wasteful allocation scheme related to the RAM cache ?
-- Leif
Re: memory management enhancement in TS-1006
Posted by Yunkai Zhang <yu...@gmail.com>.
On Mon, Dec 10, 2012 at 2:02 PM, Aleksey Kluchnikov <al...@ximad.com> wrote:
> * ming.zym@gmail.com <mi...@gmail.com> [2012-12-09 00:47:51 +0800]:
>
>
> We use ATS on freebsd (6Gb RAM) for tens millions requests at day.
> We have to reboot it twise at day becouse of memmory waste.
>
> Now we work on migration to haproxy.
> It will be great if ATS resolve memmory issues, but when it will be done?
These patches are only tested on our linux server, it seems to work
well at this time, but we are going to test it with more machines and
for more time.
Maybe we should do some compatible work on freebsd, feel free to
report the problems to us if you find any compatible issue when using
it on freebsd or other OS.
>
>
>> laster week, we have submited the patches for TS-1006,
>> https://issues.apache.org/jira/browse/TS-1006 .
>>
>> the memory issue is the most critical issue here as we are running TS in
>> heavy duty, here is the 2nd version of our memory patch in TS-1006:
>> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
>> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>>
>> if you have some issue with memory problem, please try it and feed back.
>>
>> and if you are interesting in the detail and here is the docs for the
>> detail, but in Chinese:
>> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
>> these patches done by Zhang Yunkai, guided by Chen Bin. the first
>> version contributed by Chen Bin(kuotai), and not intend to commit.
>>
>> there is another memory issue in the ram_cache, where we'd like to fix
>> it if we have TS-1006 accepted, maybe we can put our interim solution
>> later.
>>
>> thanks
>> --
>> Zhao Yongming
>> aka: zym @ apache.org, yonghao @ taobao.com
>
> --
> Aleksey Kluchnikov,
> XiMAD server side engineer (Ulyanovsk)
> Email/Jabber: alexey.kluchnikov<at>ximad.com
> Skype: kluchnikov<dot>aleksey
--
Yunkai Zhang
Work at Taobao
Re: Freebsd compiling (was: memory management enhancement in TS-1006)
Posted by Aleksey Kluchnikov <al...@ximad.com>.
* Igor Gali?? <i....@brainsware.org> [2012-12-19 08:28:25 +0000]:
>
>
> ----- Original Message -----
> > * James Peach <jp...@apache.org> [2012-12-18 22:01:28 -0800]:
> >
> > > On 18/12/2012, at 9:53 PM, Aleksey Kluchnikov <al...@ximad.com>
> > > wrote:
> > >
> > > I expect that these should apply to master. Pull the current
> > > development tree using "git clone
> > > git://git.apache.org/trafficserver.git", and apply onto that. I
> > > think you are almost there!
> > >
> >
> > Tnx. Patch is successful with development tree.
> > But i get get compiling errors under my freebsd 9.1
> >
> >
> > git clone ..
> > patch < ...
> >
> > From README:
> > autoreconf -i
> > ./configure
> >
> >
> > Get compile error.
> >
> > Without pathes:
> > ]>make
> > .... skiped
> > CXX MatcherUtils.lo
> > CXX MimeTable.lo
> > CXX MMH.lo
> > CXX ParseRules.lo
> > ParseRules.cc:34:27: error: ParseRulesCType: No such file or
> > directory
> > ParseRules.cc:37:34: error: ParseRulesCTypeToUpper: No such file or
> > directory
> > ParseRules.cc:40:34: error: ParseRulesCTypeToLower: No such file or
> > directory
> > *** [ParseRules.lo] Error code 1
>
> We require GNU make for this to work correctly.
]>gmake -v
GNU Make 3.82
]>gmake
And compile is ok.
With patches:
]>gmake
Making all in proxy/api/ts
gmake[1]: Entering directory `/usr/home/alexs/work/ATS/a/proxy/api/ts'
gmake[1]: Nothing to be done for `all'.
gmake[1]: Leaving directory `/usr/home/alexs/work/ATS/a/proxy/api/ts'
Making all in iocore
gmake[1]: Entering directory `/usr/home/alexs/work/ATS/a/iocore'
Making all in eventsystem
gmake[2]: Entering directory
`/usr/home/alexs/work/ATS/a/iocore/eventsystem'
CXX EventSystem.o
In file included from ../../lib/ts/libts.h:68,
from P_EventSystem.h:39,
from EventSystem.cc:31:
../../lib/ts/ink_queue.h:42:22: error: ink_list.h: No such file or
directory
In file included from ../../lib/ts/libts.h:68,
from P_EventSystem.h:39,
from EventSystem.cc:31:
../../lib/ts/ink_queue.h:159: error: 'ink_list' does not name a type
../../lib/ts/ink_queue.h:186: error: 'ink_list' does not name a type
gmake[2]: *** [EventSystem.o] Error 1
gmake[2]: Leaving directory `/usr/home/alexs/work/ATS/a/iocore/eventsystem'
gmake[1]: *** [all-recursive] Error 1
gmake[1]: Leaving directory `/usr/home/alexs/work/ATS/a/iocore'
gmake: *** [all-recursive] Error 1
>
> > With patches:
> > ]>make
> > Making all in proxy/api/ts
> > Making all in iocore
> > Making all in eventsystem
> > CXX EventSystem.o
> > In file included from ../../lib/ts/libts.h:68,
> > from P_EventSystem.h:39,
> > from EventSystem.cc:31:
> > ../../lib/ts/ink_queue.h:42:22: error: ink_list.h: No such file or
> > directory
> > In file included from ../../lib/ts/libts.h:68,
> > from P_EventSystem.h:39,
> > from EventSystem.cc:31:
> > ../../lib/ts/ink_queue.h:159: error: 'ink_list' does not name a type
> > ../../lib/ts/ink_queue.h:186: error: 'ink_list' does not name a type
> > *** [EventSystem.o] Error code 1
> >
> > Stop in /usr/home/alexs/work/ATS/a/iocore/eventsystem.
> > *** [all-recursive] Error code 1
> >
> > Stop in /usr/home/alexs/work/ATS/a/iocore.
> > *** [all-recursive] Error code 1
> >
> > Stop in /usr/home/alexs/work/ATS/a.
> >
> >
> > --
> > Aleksey Kluchnikov
> >
>
> --
> Igor Gali??
>
> Tel: +43 (0) 664 886 22 883
> Mail: i.galic@brainsware.org
> URL: http://brainsware.org/
> GPG: 6880 4155 74BD FD7C B515 2EA5 4B1D 9E08 A097 C9AE
--
Aleksey Kluchnikov
Re: Freebsd compiling (was: memory management enhancement in
TS-1006)
Posted by Igor Galić <i....@brainsware.org>.
----- Original Message -----
> * James Peach <jp...@apache.org> [2012-12-18 22:01:28 -0800]:
>
> > On 18/12/2012, at 9:53 PM, Aleksey Kluchnikov <al...@ximad.com>
> > wrote:
> >
> > I expect that these should apply to master. Pull the current
> > development tree using "git clone
> > git://git.apache.org/trafficserver.git", and apply onto that. I
> > think you are almost there!
> >
>
> Tnx. Patch is successful with development tree.
> But i get get compiling errors under my freebsd 9.1
>
>
> git clone ..
> patch < ...
>
> From README:
> autoreconf -i
> ./configure
>
>
> Get compile error.
>
> Without pathes:
> ]>make
> .... skiped
> CXX MatcherUtils.lo
> CXX MimeTable.lo
> CXX MMH.lo
> CXX ParseRules.lo
> ParseRules.cc:34:27: error: ParseRulesCType: No such file or
> directory
> ParseRules.cc:37:34: error: ParseRulesCTypeToUpper: No such file or
> directory
> ParseRules.cc:40:34: error: ParseRulesCTypeToLower: No such file or
> directory
> *** [ParseRules.lo] Error code 1
We require GNU make for this to work correctly.
> With patches:
> ]>make
> Making all in proxy/api/ts
> Making all in iocore
> Making all in eventsystem
> CXX EventSystem.o
> In file included from ../../lib/ts/libts.h:68,
> from P_EventSystem.h:39,
> from EventSystem.cc:31:
> ../../lib/ts/ink_queue.h:42:22: error: ink_list.h: No such file or
> directory
> In file included from ../../lib/ts/libts.h:68,
> from P_EventSystem.h:39,
> from EventSystem.cc:31:
> ../../lib/ts/ink_queue.h:159: error: 'ink_list' does not name a type
> ../../lib/ts/ink_queue.h:186: error: 'ink_list' does not name a type
> *** [EventSystem.o] Error code 1
>
> Stop in /usr/home/alexs/work/ATS/a/iocore/eventsystem.
> *** [all-recursive] Error code 1
>
> Stop in /usr/home/alexs/work/ATS/a/iocore.
> *** [all-recursive] Error code 1
>
> Stop in /usr/home/alexs/work/ATS/a.
>
>
> --
> Aleksey Kluchnikov
>
--
Igor Galić
Tel: +43 (0) 664 886 22 883
Mail: i.galic@brainsware.org
URL: http://brainsware.org/
GPG: 6880 4155 74BD FD7C B515 2EA5 4B1D 9E08 A097 C9AE
Freebsd compiling (was: memory management enhancement in TS-1006)
Posted by Aleksey Kluchnikov <al...@ximad.com>.
* James Peach <jp...@apache.org> [2012-12-18 22:01:28 -0800]:
> On 18/12/2012, at 9:53 PM, Aleksey Kluchnikov <al...@ximad.com> wrote:
>
> I expect that these should apply to master. Pull the current development tree using "git clone git://git.apache.org/trafficserver.git", and apply onto that. I think you are almost there!
>
Tnx. Patch is successful with development tree.
But i get get compiling errors under my freebsd 9.1
git clone ..
patch < ...
>From README:
autoreconf -i
./configure
Get compile error.
Without pathes:
]>make
.... skiped
CXX MatcherUtils.lo
CXX MimeTable.lo
CXX MMH.lo
CXX ParseRules.lo
ParseRules.cc:34:27: error: ParseRulesCType: No such file or directory
ParseRules.cc:37:34: error: ParseRulesCTypeToUpper: No such file or
directory
ParseRules.cc:40:34: error: ParseRulesCTypeToLower: No such file or
directory
*** [ParseRules.lo] Error code 1
With patches:
]>make
Making all in proxy/api/ts
Making all in iocore
Making all in eventsystem
CXX EventSystem.o
In file included from ../../lib/ts/libts.h:68,
from P_EventSystem.h:39,
from EventSystem.cc:31:
../../lib/ts/ink_queue.h:42:22: error: ink_list.h: No such file or
directory
In file included from ../../lib/ts/libts.h:68,
from P_EventSystem.h:39,
from EventSystem.cc:31:
../../lib/ts/ink_queue.h:159: error: 'ink_list' does not name a type
../../lib/ts/ink_queue.h:186: error: 'ink_list' does not name a type
*** [EventSystem.o] Error code 1
Stop in /usr/home/alexs/work/ATS/a/iocore/eventsystem.
*** [all-recursive] Error code 1
Stop in /usr/home/alexs/work/ATS/a/iocore.
*** [all-recursive] Error code 1
Stop in /usr/home/alexs/work/ATS/a.
--
Aleksey Kluchnikov
Re: memory management enhancement in TS-1006
Posted by James Peach <jp...@apache.org>.
On 18/12/2012, at 9:53 PM, Aleksey Kluchnikov <al...@ximad.com> wrote:
> * James Peach <jp...@apache.org> [2012-12-09 22:35:03 -0800]:
>
> How to apply this patches?
> I get last TS,
> (http://www.sai.msu.su/apache/trafficserver/trafficserver-3.2.0.tar.bz2)
>
> Try to apply pataches but get "failed" in ink_queue.cc
> Patching file a/lib/ts/ink_queue.cc using Plan A...
> Hunk #1 succeeded at 40.
> Hunk #2 succeeded at 50.
> Hunk #3 succeeded at 59.
> Hunk #4 failed at 70.
> Hunk #5 failed at 285.
> Hunk #6 failed at 634.
> Hunk #7 succeeded at 718.
> Hunk #8 succeeded at 840.
>
> Is here some howto?
I expect that these should apply to master. Pull the current development tree using "git clone git://git.apache.org/trafficserver.git", and apply onto that. I think you are almost there!
>
>> On 09/12/2012, at 10:02 PM, Aleksey Kluchnikov <al...@ximad.com> wrote:
>>
>>> * ming.zym@gmail.com <mi...@gmail.com> [2012-12-09 00:47:51 +0800]:
>>>
>>>
>>> We use ATS on freebsd (6Gb RAM) for tens millions requests at day.
>>> We have to reboot it twise at day becouse of memmory waste.
>>
>> Can you test the proposed patch and see whether it helps in your environment?
>>
>>> Now we work on migration to haproxy.
>>> It will be great if ATS resolve memmory issues, but when it will be done?
>>>
>>>
>>>> laster week, we have submited the patches for TS-1006,
>>>> https://issues.apache.org/jira/browse/TS-1006 .
>>>>
>>>> the memory issue is the most critical issue here as we are running TS in
>>>> heavy duty, here is the 2nd version of our memory patch in TS-1006:
>>>> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
>>>> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>>>>
>>>> if you have some issue with memory problem, please try it and feed back.
>>>>
>>>> and if you are interesting in the detail and here is the docs for the
>>>> detail, but in Chinese:
>>>> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
>>>> these patches done by Zhang Yunkai, guided by Chen Bin. the first
>>>> version contributed by Chen Bin(kuotai), and not intend to commit.
>>>>
>>>> there is another memory issue in the ram_cache, where we'd like to fix
>>>> it if we have TS-1006 accepted, maybe we can put our interim solution
>>>> later.
>>>>
>>>> thanks
>>>> --
>>>> Zhao Yongming
>>>> aka: zym @ apache.org, yonghao @ taobao.com
>>>
>>> --
>>> Aleksey Kluchnikov,
>>> XiMAD server side engineer (Ulyanovsk)
>>> Email/Jabber: alexey.kluchnikov<at>ximad.com
>>> Skype: kluchnikov<dot>aleksey
>
> --
> Aleksey Kluchnikov
Re: memory management enhancement in TS-1006
Posted by Aleksey Kluchnikov <al...@ximad.com>.
* James Peach <jp...@apache.org> [2012-12-09 22:35:03 -0800]:
How to apply this patches?
I get last TS,
(http://www.sai.msu.su/apache/trafficserver/trafficserver-3.2.0.tar.bz2)
Try to apply pataches but get "failed" in ink_queue.cc
Patching file a/lib/ts/ink_queue.cc using Plan A...
Hunk #1 succeeded at 40.
Hunk #2 succeeded at 50.
Hunk #3 succeeded at 59.
Hunk #4 failed at 70.
Hunk #5 failed at 285.
Hunk #6 failed at 634.
Hunk #7 succeeded at 718.
Hunk #8 succeeded at 840.
Is here some howto?
> On 09/12/2012, at 10:02 PM, Aleksey Kluchnikov <al...@ximad.com> wrote:
>
> > * ming.zym@gmail.com <mi...@gmail.com> [2012-12-09 00:47:51 +0800]:
> >
> >
> > We use ATS on freebsd (6Gb RAM) for tens millions requests at day.
> > We have to reboot it twise at day becouse of memmory waste.
>
> Can you test the proposed patch and see whether it helps in your environment?
>
> > Now we work on migration to haproxy.
> > It will be great if ATS resolve memmory issues, but when it will be done?
> >
> >
> >> laster week, we have submited the patches for TS-1006,
> >> https://issues.apache.org/jira/browse/TS-1006 .
> >>
> >> the memory issue is the most critical issue here as we are running TS in
> >> heavy duty, here is the 2nd version of our memory patch in TS-1006:
> >> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
> >> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
> >>
> >> if you have some issue with memory problem, please try it and feed back.
> >>
> >> and if you are interesting in the detail and here is the docs for the
> >> detail, but in Chinese:
> >> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
> >> these patches done by Zhang Yunkai, guided by Chen Bin. the first
> >> version contributed by Chen Bin(kuotai), and not intend to commit.
> >>
> >> there is another memory issue in the ram_cache, where we'd like to fix
> >> it if we have TS-1006 accepted, maybe we can put our interim solution
> >> later.
> >>
> >> thanks
> >> --
> >> Zhao Yongming
> >> aka: zym @ apache.org, yonghao @ taobao.com
> >
> > --
> > Aleksey Kluchnikov,
> > XiMAD server side engineer (Ulyanovsk)
> > Email/Jabber: alexey.kluchnikov<at>ximad.com
> > Skype: kluchnikov<dot>aleksey
--
Aleksey Kluchnikov
Re: memory management enhancement in TS-1006
Posted by James Peach <jp...@apache.org>.
On 09/12/2012, at 10:02 PM, Aleksey Kluchnikov <al...@ximad.com> wrote:
> * ming.zym@gmail.com <mi...@gmail.com> [2012-12-09 00:47:51 +0800]:
>
>
> We use ATS on freebsd (6Gb RAM) for tens millions requests at day.
> We have to reboot it twise at day becouse of memmory waste.
Can you test the proposed patch and see whether it helps in your environment?
> Now we work on migration to haproxy.
> It will be great if ATS resolve memmory issues, but when it will be done?
>
>
>> laster week, we have submited the patches for TS-1006,
>> https://issues.apache.org/jira/browse/TS-1006 .
>>
>> the memory issue is the most critical issue here as we are running TS in
>> heavy duty, here is the 2nd version of our memory patch in TS-1006:
>> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
>> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>>
>> if you have some issue with memory problem, please try it and feed back.
>>
>> and if you are interesting in the detail and here is the docs for the
>> detail, but in Chinese:
>> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
>> these patches done by Zhang Yunkai, guided by Chen Bin. the first
>> version contributed by Chen Bin(kuotai), and not intend to commit.
>>
>> there is another memory issue in the ram_cache, where we'd like to fix
>> it if we have TS-1006 accepted, maybe we can put our interim solution
>> later.
>>
>> thanks
>> --
>> Zhao Yongming
>> aka: zym @ apache.org, yonghao @ taobao.com
>
> --
> Aleksey Kluchnikov,
> XiMAD server side engineer (Ulyanovsk)
> Email/Jabber: alexey.kluchnikov<at>ximad.com
> Skype: kluchnikov<dot>aleksey
Re: memory management enhancement in TS-1006
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
Monday, December 10, 2012, 9:05:48 AM, you wrote:
> On 12/9/12 11:02 PM, Aleksey Kluchnikov wrote:
>> * ming.zym@gmail.com <mi...@gmail.com> [2012-12-09 00:47:51 +0800]:
>> We use ATS on freebsd (6Gb RAM) for tens millions requests at day.
>> We have to reboot it twise at day becouse of memmory waste.
Could this be TS-1580? That's a one line patch he could try.
Re: memory management enhancement in TS-1006
Posted by "ming.zym@gmail.com" <mi...@gmail.com>.
let us recall what we are going to solve in TS-1006, we are going to
solve the memory shift problem in CLFUS ram cache，let us assume we have
10G ram total, and we set ram cache for 8G, which is reasonable for not
so busy site. and we set ram cache cut size to 128KB, that is we hold 4K
8K 16K 32K 64K 128K objects in the ram cache.
in the first, all objects (size 4-128K) is able to stay in the ram cache
cause the ram cache is not full
when it is full, let us assume:
4K consumes 1G
...
128K consomes 3G
and as the time ticks, CLFUS will try to free big objects, and replace
with small objects. where the free is free to freelist, and replace is a
alloctor new 4K memory blocks. at the end, we get:
4K = 3G
...
128K = 1G
in our origin system, there is 2G memory waste, that is where the memory
gone.
then comes the root cause here, CFLUS will try to free big blocks, which
is not free indeed, it is linked into the freelist, which is still in
the TS memory. we should release those memory as we DO NOT need it
anymore, in any time, that is what we need to archive in Yunkai's patch.
FYI
在 2012-12-10一的 08:05 -0700，Leif Hedstrom写道：
> On 12/9/12 11:02 PM, Aleksey Kluchnikov wrote:
> > * ming.zym@gmail.com <mi...@gmail.com> [2012-12-09 00:47:51 +0800]:
> >
> >
> > We use ATS on freebsd (6Gb RAM) for tens millions requests at day.
> > We have to reboot it twise at day becouse of memmory waste.
>
> That sounds more like a memory leak. What the Taobao patch tries to solve is
> the case where if you have a very large spike in requests / connections,
> memory usage will go up to handle that, but then never releases that memory.
>
> One thing that I still don't understand is how this actually helps a whole
> lot. For example, if the system has to allocate 10GB of RAM to handle a
> traffic spike, and this patch later reduces that to 6GB, what are you using
> that "freed" 4GB of RAM for? You can't allocate it for something else if you
> ever want to handle that same size spike that brought you up to 10GB.
>
> What would make sense is to change how we throttle on connections, such that
> we don't let the freelist's go above a certain thresshold even at spikes.
> But, then again, I'm probably missing something here...
>
> -- Leif
>
Re: memory management enhancement in TS-1006
Posted by Leif Hedstrom <zw...@apache.org>.
On 12/9/12 11:02 PM, Aleksey Kluchnikov wrote:
> * ming.zym@gmail.com <mi...@gmail.com> [2012-12-09 00:47:51 +0800]:
>
>
> We use ATS on freebsd (6Gb RAM) for tens millions requests at day.
> We have to reboot it twise at day becouse of memmory waste.
That sounds more like a memory leak. What the Taobao patch tries to solve is
the case where if you have a very large spike in requests / connections,
memory usage will go up to handle that, but then never releases that memory.
One thing that I still don't understand is how this actually helps a whole
lot. For example, if the system has to allocate 10GB of RAM to handle a
traffic spike, and this patch later reduces that to 6GB, what are you using
that "freed" 4GB of RAM for? You can't allocate it for something else if you
ever want to handle that same size spike that brought you up to 10GB.
What would make sense is to change how we throttle on connections, such that
we don't let the freelist's go above a certain thresshold even at spikes.
But, then again, I'm probably missing something here...
-- Leif
Re: memory management enhancement in TS-1006
Posted by Aleksey Kluchnikov <al...@ximad.com>.
* ming.zym@gmail.com <mi...@gmail.com> [2012-12-09 00:47:51 +0800]:
We use ATS on freebsd (6Gb RAM) for tens millions requests at day.
We have to reboot it twise at day becouse of memmory waste.
Now we work on migration to haproxy.
It will be great if ATS resolve memmory issues, but when it will be done?
> laster week, we have submited the patches for TS-1006,
> https://issues.apache.org/jira/browse/TS-1006 .
>
> the memory issue is the most critical issue here as we are running TS in
> heavy duty, here is the 2nd version of our memory patch in TS-1006:
> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>
> if you have some issue with memory problem, please try it and feed back.
>
> and if you are interesting in the detail and here is the docs for the
> detail, but in Chinese:
> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
> these patches done by Zhang Yunkai, guided by Chen Bin. the first
> version contributed by Chen Bin(kuotai), and not intend to commit.
>
> there is another memory issue in the ram_cache, where we'd like to fix
> it if we have TS-1006 accepted, maybe we can put our interim solution
> later.
>
> thanks
> --
> Zhao Yongming
> aka: zym @ apache.org, yonghao @ taobao.com
--
Aleksey Kluchnikov,
XiMAD server side engineer (Ulyanovsk)
Email/Jabber: alexey.kluchnikov<at>ximad.com
Skype: kluchnikov<dot>aleksey
Re: memory management enhancement in TS-1006
Posted by Yunkai Zhang <yu...@gmail.com>.
On Mon, Dec 10, 2012 at 6:48 AM, James Peach <jp...@apache.org> wrote:
> On 08/12/2012, at 8:47 AM, ming.zym@gmail.com wrote:
>
>> laster week, we have submited the patches for TS-1006,
>> https://issues.apache.org/jira/browse/TS-1006 .
>>
>> the memory issue is the most critical issue here as we are running TS in
>> heavy duty, here is the 2nd version of our memory patch in TS-1006:
>> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
>> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>>
>> if you have some issue with memory problem, please try it and feed back.
>>
>> and if you are interesting in the detail and here is the docs for the
>> detail, but in Chinese:
>> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
>> these patches done by Zhang Yunkai, guided by Chen Bin. the first
>> version contributed by Chen Bin(kuotai), and not intend to commit.
>
> I made a pass at translating this with the help of google and bing. If anyone else wants to pitch in, feel free :)
>
> https://docs.google.com/document/d/1bT5nMJiN9gIC_B5ffiHv_-dsFzcXSkEGn7Dx1Em3huA/edit#
Thanks:)
>
>
>>
>> there is another memory issue in the ram_cache, where we'd like to fix
>> it if we have TS-1006 accepted, maybe we can put our interim solution
>> later.
>>
>> thanks
>> --
>> Zhao Yongming
>> aka: zym @ apache.org, yonghao @ taobao.com
>>
>
--
Yunkai Zhang
Work at Taobao
Re: memory management enhancement in TS-1006
Posted by Yunkai Zhang <yu...@gmail.com>.
On Mon, Dec 10, 2012 at 6:48 AM, James Peach <jp...@apache.org> wrote:
> On 08/12/2012, at 8:47 AM, ming.zym@gmail.com wrote:
>
>> laster week, we have submited the patches for TS-1006,
>> https://issues.apache.org/jira/browse/TS-1006 .
>>
>> the memory issue is the most critical issue here as we are running TS in
>> heavy duty, here is the 2nd version of our memory patch in TS-1006:
>> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
>> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>>
>> if you have some issue with memory problem, please try it and feed back.
>>
>> and if you are interesting in the detail and here is the docs for the
>> detail, but in Chinese:
>> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
>> these patches done by Zhang Yunkai, guided by Chen Bin. the first
>> version contributed by Chen Bin(kuotai), and not intend to commit.
>
> I made a pass at translating this with the help of google and bing. If anyone else wants to pitch in, feel free :)
>
> https://docs.google.com/document/d/1bT5nMJiN9gIC_B5ffiHv_-dsFzcXSkEGn7Dx1Em3huA/edit#
Thanks:)
>
>
>>
>> there is another memory issue in the ram_cache, where we'd like to fix
>> it if we have TS-1006 accepted, maybe we can put our interim solution
>> later.
>>
>> thanks
>> --
>> Zhao Yongming
>> aka: zym @ apache.org, yonghao @ taobao.com
>>
>
--
Yunkai Zhang
Work at Taobao
Re: memory management enhancement in TS-1006
Posted by James Peach <jp...@apache.org>.
On 08/12/2012, at 8:47 AM, ming.zym@gmail.com wrote:
> laster week, we have submited the patches for TS-1006,
> https://issues.apache.org/jira/browse/TS-1006 .
>
> the memory issue is the most critical issue here as we are running TS in
> heavy duty, here is the 2nd version of our memory patch in TS-1006:
> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>
> if you have some issue with memory problem, please try it and feed back.
>
> and if you are interesting in the detail and here is the docs for the
> detail, but in Chinese:
> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
> these patches done by Zhang Yunkai, guided by Chen Bin. the first
> version contributed by Chen Bin(kuotai), and not intend to commit.
I made a pass at translating this with the help of google and bing. If anyone else wants to pitch in, feel free :)
https://docs.google.com/document/d/1bT5nMJiN9gIC_B5ffiHv_-dsFzcXSkEGn7Dx1Em3huA/edit#
>
> there is another memory issue in the ram_cache, where we'd like to fix
> it if we have TS-1006 accepted, maybe we can put our interim solution
> later.
>
> thanks
> --
> Zhao Yongming
> aka: zym @ apache.org, yonghao @ taobao.com
>
Re: memory management enhancement in TS-1006
Posted by Leif Hedstrom <zw...@apache.org>.
On 12/8/12 9:47 AM, ming.zym@gmail.com wrote:
> laster week, we have submited the patches for TS-1006,
> https://issues.apache.org/jira/browse/TS-1006 .
Ok, I'm reading TS-1006 again, and i'll take a look at the patches. But just
to be sure, the idea here is to fix something that's an apparently "leak" or
wasteful allocation scheme related to the RAM cache ?
-- Leif
Re: memory management enhancement in TS-1006
Posted by "ming.zym@gmail.com" <mi...@gmail.com>.
no, the comments is for the following directive.
if you want to find the docs for each directive, look at the comments
above it :D
在 2012-12-30日的 14:34 +0100，Reindl Harald写道：
>
> Am 30.12.2012 14:28, schrieb ming.zym@gmail.com:
> > NOTE: you must change the LRU algorithm to Standard LRU by set
> > proxy.config.cache.ram_cache.algorithm to 1
>
> uhm this is compression?! why?
>
> CONFIG proxy.config.cache.ram_cache.algorithm INT 0
> # Compress the content of the ram cache:
> # 0 : no compression
> # 1 : fastlz (extremely fast, relatively low compression)
> # 2 : libz (moderate speed, reasonable compression)
> # 3 : liblzma (very slow, high compression)
> # NOTE: compression runs on task threads. To use more cores for
> # compression, increase proxy.config.task_threads.
>
--
Zhao Yongming
aka: zym @ apache.org, yonghao @ taobao.com
Re: memory management enhancement in TS-1006
Posted by Reindl Harald <h....@thelounge.net>.
Am 30.12.2012 14:28, schrieb ming.zym@gmail.com:
> NOTE: you must change the LRU algorithm to Standard LRU by set
> proxy.config.cache.ram_cache.algorithm to 1
uhm this is compression?! why?
CONFIG proxy.config.cache.ram_cache.algorithm INT 0
# Compress the content of the ram cache:
# 0 : no compression
# 1 : fastlz (extremely fast, relatively low compression)
# 2 : libz (moderate speed, reasonable compression)
# 3 : liblzma (very slow, high compression)
# NOTE: compression runs on task threads. To use more cores for
# compression, increase proxy.config.task_threads.
Re: memory management enhancement in TS-1006
Posted by "ming.zym@gmail.com" <mi...@gmail.com>.
after the 6th patch, we have all our codes done, we'd like to stop and
get all the review done. this is for the first attempt to the memory
core, not the last :D
let us sum up what is there:
1, we made a dyn-freelist, which will prefer to free the memory when
some list is wasting memory.
2, we changed the direct IO buffer allocator, using the memory in LRU
list instead of the freelist, the result is POP one, read one, on the
standard LRU.
by the #2, we have strict the standard LRU to the memory blocks when the
ram cache is full. by #1, we can free the unused memory buckets into the
system or reuse internally.
that is what we have done in Taobao, which is very critical enhancement
for us, or ATS is not usable for us at all.
NOTE: you must change the LRU algorithm to Standard LRU by set
proxy.config.cache.ram_cache.algorithm to 1
what we have done is hard to adapt into CLFUS, we are still seeking the
way for CLFUS. Anyway, let us make sure we can lock the memory down.
thanks
在 2012-12-09日的 00:47 +0800，ming.zym@gmail.com写道：
> laster week, we have submited the patches for TS-1006,
> https://issues.apache.org/jira/browse/TS-1006 .
>
> the memory issue is the most critical issue here as we are running TS in
> heavy duty, here is the 2nd version of our memory patch in TS-1006:
> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>
> if you have some issue with memory problem, please try it and feed back.
>
> and if you are interesting in the detail and here is the docs for the
> detail, but in Chinese:
> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
> these patches done by Zhang Yunkai, guided by Chen Bin. the first
> version contributed by Chen Bin(kuotai), and not intend to commit.
>
> there is another memory issue in the ram_cache, where we'd like to fix
> it if we have TS-1006 accepted, maybe we can put our interim solution
> later.
>
> thanks
--
Zhao Yongming
aka: zym @ apache.org, yonghao @ taobao.com
Re: memory management enhancement in TS-1006
Posted by James Peach <jp...@apache.org>.
On 08/12/2012, at 8:47 AM, ming.zym@gmail.com wrote:
> laster week, we have submited the patches for TS-1006,
> https://issues.apache.org/jira/browse/TS-1006 .
>
> the memory issue is the most critical issue here as we are running TS in
> heavy duty, here is the 2nd version of our memory patch in TS-1006:
> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>
> if you have some issue with memory problem, please try it and feed back.
>
> and if you are interesting in the detail and here is the docs for the
> detail, but in Chinese:
> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
> these patches done by Zhang Yunkai, guided by Chen Bin. the first
> version contributed by Chen Bin(kuotai), and not intend to commit.
I made a pass at translating this with the help of google and bing. If anyone else wants to pitch in, feel free :)
https://docs.google.com/document/d/1bT5nMJiN9gIC_B5ffiHv_-dsFzcXSkEGn7Dx1Em3huA/edit#
>
> there is another memory issue in the ram_cache, where we'd like to fix
> it if we have TS-1006 accepted, maybe we can put our interim solution
> later.
>
> thanks
> --
> Zhao Yongming
> aka: zym @ apache.org, yonghao @ taobao.com
>
Re: memory management enhancement in TS-1006
Posted by "ming.zym@gmail.com" <mi...@gmail.com>.
after the 6th patch, we have all our codes done, we'd like to stop and
get all the review done. this is for the first attempt to the memory
core, not the last :D
let us sum up what is there:
1, we made a dyn-freelist, which will prefer to free the memory when
some list is wasting memory.
2, we changed the direct IO buffer allocator, using the memory in LRU
list instead of the freelist, the result is POP one, read one, on the
standard LRU.
by the #2, we have strict the standard LRU to the memory blocks when the
ram cache is full. by #1, we can free the unused memory buckets into the
system or reuse internally.
that is what we have done in Taobao, which is very critical enhancement
for us, or ATS is not usable for us at all.
NOTE: you must change the LRU algorithm to Standard LRU by set
proxy.config.cache.ram_cache.algorithm to 1
what we have done is hard to adapt into CLFUS, we are still seeking the
way for CLFUS. Anyway, let us make sure we can lock the memory down.
thanks
在 2012-12-09日的 00:47 +0800，ming.zym@gmail.com写道：
> laster week, we have submited the patches for TS-1006,
> https://issues.apache.org/jira/browse/TS-1006 .
>
> the memory issue is the most critical issue here as we are running TS in
> heavy duty, here is the 2nd version of our memory patch in TS-1006:
> https://issues.apache.org/jira/secure/attachment/12556207/0001-Allocator-optimize-InkFreeList-memory-pool.patch
> https://issues.apache.org/jira/secure/attachment/12556208/0002-Allocator-make-InkFreeList-memory-pool-configurable.patch
>
> if you have some issue with memory problem, please try it and feed back.
>
> and if you are interesting in the detail and here is the docs for the
> detail, but in Chinese:
> https://docs.google.com/document/d/12ojZqZdPH-Xuhjo_UgX9i5o9jEO3JRdB577g1Azhut4/edit
> these patches done by Zhang Yunkai, guided by Chen Bin. the first
> version contributed by Chen Bin(kuotai), and not intend to commit.
>
> there is another memory issue in the ram_cache, where we'd like to fix
> it if we have TS-1006 accepted, maybe we can put our interim solution
> later.
>
> thanks
--
Zhao Yongming
aka: zym @ apache.org, yonghao @ taobao.com