[jira] [Assigned] (TS-4266) ATS memory statistics shows that memory utilization is doubled after “traffic_ctlconfig reload”. And it is failed as it cannot find enough memory.

["Kit Chan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/TS-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kit Chan reassigned TS-4266:
----------------------------

    Assignee: Kit Chan

> ATS memory statistics shows that memory utilization is doubled after “traffic_ctlconfig reload”. And it is failed as it cannot find enough memory.
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: TS-4266
>                 URL: https://issues.apache.org/jira/browse/TS-4266
>             Project: Traffic Server
>          Issue Type: Bug
>          Components: Lua
>            Reporter: Rajendra Kishore Bonumahanti
>            Assignee: Kit Chan
>             Fix For: sometime
>
>
> ATS memory statistics shows memory utilization is doubled after “traffic_ctl config reload”. We get “not enough memory” error in the subsequent attempt and “config reload” fails.
> ATS is configured with 100 map entries in remap.config, all share the same lua script.
> ATS is started: The memory information is..
> [root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
> total kB         1416092  670256  663736
> After 1st Config reload:
> [root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
> total kB         1932660 1128084 1121544
> After 2nd config reload: It had failed with error “not enough memory” and memory status as..
> [root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
> total kB         2170756 1167808 1160836
> Error displayed in diags.log:
> =======================
> [Mar  8 23:27:27.580] Server {0x2af92498b700} WARNING: Failed to create new instance for plugin /opt/trafficserver/libexec/trafficserver/tslua.so (not a TS_SUCCESS return)
> [Mar  8 23:27:27.580] Server {0x2af92498b700} WARNING: Could not add rule at line #3; Aborting!
> [Mar  8 23:27:27.580] Server {0x2af92498b700} WARNING: [ReverseProxy] Can't create new remap instance for plugin "/opt/trafficserver/libexec/trafficserver/tslua.so" - [ts_lua_add_module] luaL_loadfile /opt/trafficserver/etc/trafficserver/lua/process_remap.lua failed: not enough memory at line 3
> [Mar  8 23:27:27.580] Server {0x2af92498b700} WARNING: something failed during BuildTable() -- check your remap plugins!
> [Mar  8 23:27:27.595] Server {0x2af92498b700} WARNING: failed to reload remap.config, not replacing!
> Lua VM memory size at that time ,ts.debug(FUNCTION..'Lua VM memory: '..collectgarbage("count")) 
> [Mar  8 23:27:27.579] Server {0x2af92498b700} DIAG: (ts_lua) __init__(): Lua VM memory: 3629.7060546875
> This shows that Lua VMs are hitting the max capacity.
> Solution:
> =======
> I looked at the ts_lua code TSRemapDeleteInstance () [ts_lua.c ] and ts_lua_del_module() [ts_lua_util.c] which does cleaning of the lua memory for the instance. However the lua memory is not released and reused. 
> So, I have added code to start the garbage collector in ts_lua_del_module() .
> int
> ts_lua_del_module(ts_lua_instance_conf *conf, ts_lua_main_ctx *arr, int n)
> {
> ….
>     lua_newtable(L);
>     lua_replace(L, LUA_GLOBALSINDEX); /* L[GLOBAL] = EMPTY  */
>     lua_gc(L, LUA_GCCOLLECT, 0);
>     TSMutexUnlock(arr[i].mutexp);
> }
>   return 0;
> }
> This has improved the situation. However, I also added garbage collection in ts_lua_add_module() at the end. With these two additions, we have tested the code, the memory utilization is stable and we could do config reload at lest 100 times with the background load.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)