[jira] [Created] (TS-4266) ATS memory statistics shows that memory utilization is doubled after “traffic_ctlconfig reload”. And it is failed as it cannot find enough memory.

["Rajendra Kishore Bonumahanti (JIRA)" <ji...@apache.org>]

Rajendra Kishore Bonumahanti created TS-4266:
------------------------------------------------

             Summary: ATS memory statistics shows that memory utilization is doubled after “traffic_ctlconfig reload”. And it is failed as it cannot find enough memory.
                 Key: TS-4266
                 URL: https://issues.apache.org/jira/browse/TS-4266
             Project: Traffic Server
          Issue Type: Bug
          Components: Lua
            Reporter: Rajendra Kishore Bonumahanti


ATS memory statistics shows memory utilization is doubled after “traffic_ctl config reload”. We get “not enough memory” error in the subsequent attempt and “config reload” fails.

ATS is configured with 100 map entries in remap.config, all share the same lua script.

ATS is started: The memory information is..
[root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
total kB         1416092  670256  663736

After 1st Config reload:
[root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
total kB         1932660 1128084 1121544

After 2nd config reload: It had failed with error “not enough memory” and memory status as..
[root@mtanjv8cdnc73 trafficserver]# pmap -x 113330 | grep total
total kB         2170756 1167808 1160836

Error displayed in diags.log:
=======================
[Mar  8 23:27:27.580] Server {0x2af92498b700} WARNING: Failed to create new instance for plugin /opt/trafficserver/libexec/trafficserver/tslua.so (not a TS_SUCCESS return)
[Mar  8 23:27:27.580] Server {0x2af92498b700} WARNING: Could not add rule at line #3; Aborting!
[Mar  8 23:27:27.580] Server {0x2af92498b700} WARNING: [ReverseProxy] Can't create new remap instance for plugin "/opt/trafficserver/libexec/trafficserver/tslua.so" - [ts_lua_add_module] luaL_loadfile /opt/trafficserver/etc/trafficserver/lua/process_remap.lua failed: not enough memory at line 3
[Mar  8 23:27:27.580] Server {0x2af92498b700} WARNING: something failed during BuildTable() -- check your remap plugins!
[Mar  8 23:27:27.595] Server {0x2af92498b700} WARNING: failed to reload remap.config, not replacing!

Lua VM memory size at that time ,ts.debug(FUNCTION..'Lua VM memory: '..collectgarbage("count")) 
[Mar  8 23:27:27.579] Server {0x2af92498b700} DIAG: (ts_lua) __init__(): Lua VM memory: 3629.7060546875

This shows that Lua VMs are hitting the max capacity.

Solution:
=======
I looked at the ts_lua code TSRemapDeleteInstance () [ts_lua.c ] and ts_lua_del_module() [ts_lua_util.c] which does cleaning of the lua memory for the instance. However the lua memory is not released and reused. 
So, I have added code to start the garbage collector in ts_lua_del_module() .
int
ts_lua_del_module(ts_lua_instance_conf *conf, ts_lua_main_ctx *arr, int n)
{
….
    lua_newtable(L);
    lua_replace(L, LUA_GLOBALSINDEX); /* L[GLOBAL] = EMPTY  */

    lua_gc(L, LUA_GCCOLLECT, 0);

    TSMutexUnlock(arr[i].mutexp);
}

  return 0;
}
This has improved the situation. However, I also added garbage collection in ts_lua_add_module() at the end. With these two additions, we have tested the code, the memory utilization is stable and we could do config reload at lest 100 times with the background load.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)