You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by "Chou, Peter" <pb...@labs.att.com> on 2019/04/30 18:05:22 UTC
Questions on tslua plugin and luajit.
Hi,
Few questions on LuaJIT in relation to the ts_lua plugin.
1. We did get the following core dump in production. Does this fit the signature
for a LuaJIT memory exhaust?
Program terminated with signal 11, Segmentation fault.
#0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install ATTats-7.1.4-126.el7.x86_64
(gdb) bt
#0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
#1 0x00002b67b4bfebb7 in exit () from /lib64/libc.so.6
#2 0x0000000000543d15 in lj_err_throw ()
#3 0x0000000000543f33 in lj_err_mem ()
#4 0x0000000000547e72 in lj_mem_realloc ()
#5 0x000000000055ffe5 in stack_init ()
#6 0x00000000005605b6 in lj_state_new ()
#7 0x000000000052788b in lua_newthread ()
#8 0x00002b67b7eb1cf3 in ts_lua_create_http_ctx (main_ctx=main_ctx@entry=0x8bdeab0,
conf=conf@entry=0x2b692bd5f400) at experimental/ts_lua/ts_lua_util.c:421
#9 0x00002b67b7ea8941 in ts_lua_remap_plugin_init (rri=0x4f85e50, rh=0x2b69481b0520, ih=0x2b692bd5f400)
at experimental/ts_lua/ts_lua.c:174
#10 TSRemapDoRemap (ih=0x2b692bd5f400, rh=0x2b69481b0520, rri=0x4f85e50) at experimental/ts_lua/ts_lua.c:237
2. I notice in 8.0.x forwards that the LuaJIT bundled with ATS was removed.
I saw that the LuaJIT that was bundled with ATS 7.1.x was 2.0.4 which only
supports 1GB of memory in Linux. If we use the system libluajit in conjunction
with ATS 8.0.x (under Ubuntu Linux 18.04 LTS this is 2.1.0-beta3) we should
now be able to use 2GB correct?
Thanks,
Peter
RE: Questions on tslua plugin and luajit.
Posted by "Chou, Peter" <pb...@labs.att.com>.
Kit,
I did not try the new luajit in conjunction with ATS/tslua plugin yet.
I only tested with the command line luajit command.
luajit-2.1.0-beta3 --
pebc@piplup2:~/src/luajit-2.1.0~beta3+dfsg/src$ ./luajit ~/test-mem.lua 2000
total bytes allocated = 2048000000
pebc@piplup2:~/src/luajit-2.1.0~beta3+dfsg/src$ ./luajit ~/test-mem.lua 2100
./luajit: not enough memory
pebc@piplup2:~/src/luajit-2.1.0~beta3+dfsg/src$ ./luajit -v
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2017 Mike Pall. http://luajit.org/
luajit-2.0.4 --
pebc@piplup2:~/git/codecloud/trafficserver/lib/luajit/src$ ./luajit -v
LuaJIT 2.0.4 -- Copyright (C) 2005-2015 Mike Pall. http://luajit.org/
pebc@piplup2:~/git/codecloud/trafficserver/lib/luajit/src$ ./luajit ~/test-mem.lua 1000
total bytes allocated = 1024000000
pebc@piplup2:~/git/codecloud/trafficserver/lib/luajit/src$ ./luajit ~/test-mem.lua 1100
./luajit: not enough memory
For compilation --
The 2.0.4 was from the ATS source tree.
The 2.1.0-beta3 is the version that comes with Ubuntu Linux 18.04 LTS.
I just did apt-get source libluajit-5.1-2, then cd, then make so nothing special.
If you look in lj_alloc.c I did see find a comment that is new between versions --
/* This only allows 1GB on Linux. So fallback to probing to get 2GB. */
Test program --
pebc@piplup2:~$ cat test-mem.lua
count = 0
t = {}
FH = io.open("/dev/urandom", "rb")
for i = 1, arg[1], 1 do
t[i] = FH:read(1024000)
count = count + string.len(t[1])
end
print ("total bytes allocated = " .. count)
Thanks,
Peter
-----Original Message-----
From: Shu Kit Chan <ch...@gmail.com>
Sent: Wednesday, May 1, 2019 12:47 PM
To: dev@trafficserver.apache.org
Subject: Re: Questions on tslua plugin and luajit.
Interesting.
So with luajit 2.1.0-beta3, you can get to 2GB? Or more ? Did you get
the PANIC error at the 2GB mark?
Also how do you get to know the memory usage of your Lua program?
Using "collectgarbage"?
Also how do you compile your library? is there any difference in how
you compile 2.1.0-beta3 vs 2.0.4 ?
Thanks for the info in advance.
Kit
On Wed, May 1, 2019 at 9:09 AM Chou, Peter <pb...@labs.att.com> wrote:
>
> Kit,
>
> Thanks for the response. With regards to the LuaJIT 2.1.0-beta3, I have not tried
> using the ts_lua plugin, but with a simple test program using the luajit
> command line I was able to allocate 2.0GB of memory under Linux. On the same
> machine, using the luajit compiled within the ATS tree, I was only able to
> allocate 1.0GB of memory.
>
> BTW, the test program was just opening /dev/urandom and reading 1024000 byte
> strings into a table until out of memory was reported. Hopefully, the increased
> limit translates into the plugin also.
>
> Thanks,
> Peter
>
> -----Original Message-----
> From: Shu Kit Chan <ch...@gmail.com>
> Sent: Wednesday, May 1, 2019 2:50 AM
> To: dev@trafficserver.apache.org
> Subject: Re: Questions on tslua plugin and luajit.
>
> 1) Yes. That looks like it.
> you can try to lower the number of states in your mapping rules. The
> default is 256. That is the number of lua_State we start with.
>
> 2) I am not aware that luajit-2.1.0-beta3 will have any improvement to
> the memory limit problem.
>
> Thanks.
>
> Kit
>
> On Tue, Apr 30, 2019 at 11:05 AM Chou, Peter <pb...@labs.att.com> wrote:
> >
> > Hi,
> >
> > Few questions on LuaJIT in relation to the ts_lua plugin.
> >
> > 1. We did get the following core dump in production. Does this fit the signature
> > for a LuaJIT memory exhaust?
> >
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
> > Missing separate debuginfos, use: debuginfo-install ATTats-7.1.4-126.el7.x86_64
> > (gdb) bt
> > #0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
> > #1 0x00002b67b4bfebb7 in exit () from /lib64/libc.so.6
> > #2 0x0000000000543d15 in lj_err_throw ()
> > #3 0x0000000000543f33 in lj_err_mem ()
> > #4 0x0000000000547e72 in lj_mem_realloc ()
> > #5 0x000000000055ffe5 in stack_init ()
> > #6 0x00000000005605b6 in lj_state_new ()
> > #7 0x000000000052788b in lua_newthread ()
> > #8 0x00002b67b7eb1cf3 in ts_lua_create_http_ctx (main_ctx=main_ctx@entry=0x8bdeab0,
> > conf=conf@entry=0x2b692bd5f400) at experimental/ts_lua/ts_lua_util.c:421
> > #9 0x00002b67b7ea8941 in ts_lua_remap_plugin_init (rri=0x4f85e50, rh=0x2b69481b0520, ih=0x2b692bd5f400)
> > at experimental/ts_lua/ts_lua.c:174
> > #10 TSRemapDoRemap (ih=0x2b692bd5f400, rh=0x2b69481b0520, rri=0x4f85e50) at experimental/ts_lua/ts_lua.c:237
> >
> > 2. I notice in 8.0.x forwards that the LuaJIT bundled with ATS was removed.
> > I saw that the LuaJIT that was bundled with ATS 7.1.x was 2.0.4 which only
> > supports 1GB of memory in Linux. If we use the system libluajit in conjunction
> > with ATS 8.0.x (under Ubuntu Linux 18.04 LTS this is 2.1.0-beta3) we should
> > now be able to use 2GB correct?
> >
> > Thanks,
> > Peter
Re: Questions on tslua plugin and luajit.
Posted by Shu Kit Chan <ch...@gmail.com>.
Interesting.
So with luajit 2.1.0-beta3, you can get to 2GB? Or more ? Did you get
the PANIC error at the 2GB mark?
Also how do you get to know the memory usage of your Lua program?
Using "collectgarbage"?
Also how do you compile your library? is there any difference in how
you compile 2.1.0-beta3 vs 2.0.4 ?
Thanks for the info in advance.
Kit
On Wed, May 1, 2019 at 9:09 AM Chou, Peter <pb...@labs.att.com> wrote:
>
> Kit,
>
> Thanks for the response. With regards to the LuaJIT 2.1.0-beta3, I have not tried
> using the ts_lua plugin, but with a simple test program using the luajit
> command line I was able to allocate 2.0GB of memory under Linux. On the same
> machine, using the luajit compiled within the ATS tree, I was only able to
> allocate 1.0GB of memory.
>
> BTW, the test program was just opening /dev/urandom and reading 1024000 byte
> strings into a table until out of memory was reported. Hopefully, the increased
> limit translates into the plugin also.
>
> Thanks,
> Peter
>
> -----Original Message-----
> From: Shu Kit Chan <ch...@gmail.com>
> Sent: Wednesday, May 1, 2019 2:50 AM
> To: dev@trafficserver.apache.org
> Subject: Re: Questions on tslua plugin and luajit.
>
> 1) Yes. That looks like it.
> you can try to lower the number of states in your mapping rules. The
> default is 256. That is the number of lua_State we start with.
>
> 2) I am not aware that luajit-2.1.0-beta3 will have any improvement to
> the memory limit problem.
>
> Thanks.
>
> Kit
>
> On Tue, Apr 30, 2019 at 11:05 AM Chou, Peter <pb...@labs.att.com> wrote:
> >
> > Hi,
> >
> > Few questions on LuaJIT in relation to the ts_lua plugin.
> >
> > 1. We did get the following core dump in production. Does this fit the signature
> > for a LuaJIT memory exhaust?
> >
> > Program terminated with signal 11, Segmentation fault.
> > #0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
> > Missing separate debuginfos, use: debuginfo-install ATTats-7.1.4-126.el7.x86_64
> > (gdb) bt
> > #0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
> > #1 0x00002b67b4bfebb7 in exit () from /lib64/libc.so.6
> > #2 0x0000000000543d15 in lj_err_throw ()
> > #3 0x0000000000543f33 in lj_err_mem ()
> > #4 0x0000000000547e72 in lj_mem_realloc ()
> > #5 0x000000000055ffe5 in stack_init ()
> > #6 0x00000000005605b6 in lj_state_new ()
> > #7 0x000000000052788b in lua_newthread ()
> > #8 0x00002b67b7eb1cf3 in ts_lua_create_http_ctx (main_ctx=main_ctx@entry=0x8bdeab0,
> > conf=conf@entry=0x2b692bd5f400) at experimental/ts_lua/ts_lua_util.c:421
> > #9 0x00002b67b7ea8941 in ts_lua_remap_plugin_init (rri=0x4f85e50, rh=0x2b69481b0520, ih=0x2b692bd5f400)
> > at experimental/ts_lua/ts_lua.c:174
> > #10 TSRemapDoRemap (ih=0x2b692bd5f400, rh=0x2b69481b0520, rri=0x4f85e50) at experimental/ts_lua/ts_lua.c:237
> >
> > 2. I notice in 8.0.x forwards that the LuaJIT bundled with ATS was removed.
> > I saw that the LuaJIT that was bundled with ATS 7.1.x was 2.0.4 which only
> > supports 1GB of memory in Linux. If we use the system libluajit in conjunction
> > with ATS 8.0.x (under Ubuntu Linux 18.04 LTS this is 2.1.0-beta3) we should
> > now be able to use 2GB correct?
> >
> > Thanks,
> > Peter
RE: Questions on tslua plugin and luajit.
Posted by "Chou, Peter" <pb...@labs.att.com>.
Kit,
Thanks for the response. With regards to the LuaJIT 2.1.0-beta3, I have not tried
using the ts_lua plugin, but with a simple test program using the luajit
command line I was able to allocate 2.0GB of memory under Linux. On the same
machine, using the luajit compiled within the ATS tree, I was only able to
allocate 1.0GB of memory.
BTW, the test program was just opening /dev/urandom and reading 1024000 byte
strings into a table until out of memory was reported. Hopefully, the increased
limit translates into the plugin also.
Thanks,
Peter
-----Original Message-----
From: Shu Kit Chan <ch...@gmail.com>
Sent: Wednesday, May 1, 2019 2:50 AM
To: dev@trafficserver.apache.org
Subject: Re: Questions on tslua plugin and luajit.
1) Yes. That looks like it.
you can try to lower the number of states in your mapping rules. The
default is 256. That is the number of lua_State we start with.
2) I am not aware that luajit-2.1.0-beta3 will have any improvement to
the memory limit problem.
Thanks.
Kit
On Tue, Apr 30, 2019 at 11:05 AM Chou, Peter <pb...@labs.att.com> wrote:
>
> Hi,
>
> Few questions on LuaJIT in relation to the ts_lua plugin.
>
> 1. We did get the following core dump in production. Does this fit the signature
> for a LuaJIT memory exhaust?
>
> Program terminated with signal 11, Segmentation fault.
> #0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install ATTats-7.1.4-126.el7.x86_64
> (gdb) bt
> #0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
> #1 0x00002b67b4bfebb7 in exit () from /lib64/libc.so.6
> #2 0x0000000000543d15 in lj_err_throw ()
> #3 0x0000000000543f33 in lj_err_mem ()
> #4 0x0000000000547e72 in lj_mem_realloc ()
> #5 0x000000000055ffe5 in stack_init ()
> #6 0x00000000005605b6 in lj_state_new ()
> #7 0x000000000052788b in lua_newthread ()
> #8 0x00002b67b7eb1cf3 in ts_lua_create_http_ctx (main_ctx=main_ctx@entry=0x8bdeab0,
> conf=conf@entry=0x2b692bd5f400) at experimental/ts_lua/ts_lua_util.c:421
> #9 0x00002b67b7ea8941 in ts_lua_remap_plugin_init (rri=0x4f85e50, rh=0x2b69481b0520, ih=0x2b692bd5f400)
> at experimental/ts_lua/ts_lua.c:174
> #10 TSRemapDoRemap (ih=0x2b692bd5f400, rh=0x2b69481b0520, rri=0x4f85e50) at experimental/ts_lua/ts_lua.c:237
>
> 2. I notice in 8.0.x forwards that the LuaJIT bundled with ATS was removed.
> I saw that the LuaJIT that was bundled with ATS 7.1.x was 2.0.4 which only
> supports 1GB of memory in Linux. If we use the system libluajit in conjunction
> with ATS 8.0.x (under Ubuntu Linux 18.04 LTS this is 2.1.0-beta3) we should
> now be able to use 2GB correct?
>
> Thanks,
> Peter
Re: Questions on tslua plugin and luajit.
Posted by Shu Kit Chan <ch...@gmail.com>.
1) Yes. That looks like it.
you can try to lower the number of states in your mapping rules. The
default is 256. That is the number of lua_State we start with.
2) I am not aware that luajit-2.1.0-beta3 will have any improvement to
the memory limit problem.
Thanks.
Kit
On Tue, Apr 30, 2019 at 11:05 AM Chou, Peter <pb...@labs.att.com> wrote:
>
> Hi,
>
> Few questions on LuaJIT in relation to the ts_lua plugin.
>
> 1. We did get the following core dump in production. Does this fit the signature
> for a LuaJIT memory exhaust?
>
> Program terminated with signal 11, Segmentation fault.
> #0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install ATTats-7.1.4-126.el7.x86_64
> (gdb) bt
> #0 0x00002b67b4bfeacd in __run_exit_handlers () from /lib64/libc.so.6
> #1 0x00002b67b4bfebb7 in exit () from /lib64/libc.so.6
> #2 0x0000000000543d15 in lj_err_throw ()
> #3 0x0000000000543f33 in lj_err_mem ()
> #4 0x0000000000547e72 in lj_mem_realloc ()
> #5 0x000000000055ffe5 in stack_init ()
> #6 0x00000000005605b6 in lj_state_new ()
> #7 0x000000000052788b in lua_newthread ()
> #8 0x00002b67b7eb1cf3 in ts_lua_create_http_ctx (main_ctx=main_ctx@entry=0x8bdeab0,
> conf=conf@entry=0x2b692bd5f400) at experimental/ts_lua/ts_lua_util.c:421
> #9 0x00002b67b7ea8941 in ts_lua_remap_plugin_init (rri=0x4f85e50, rh=0x2b69481b0520, ih=0x2b692bd5f400)
> at experimental/ts_lua/ts_lua.c:174
> #10 TSRemapDoRemap (ih=0x2b692bd5f400, rh=0x2b69481b0520, rri=0x4f85e50) at experimental/ts_lua/ts_lua.c:237
>
> 2. I notice in 8.0.x forwards that the LuaJIT bundled with ATS was removed.
> I saw that the LuaJIT that was bundled with ATS 7.1.x was 2.0.4 which only
> supports 1GB of memory in Linux. If we use the system libluajit in conjunction
> with ATS 8.0.x (under Ubuntu Linux 18.04 LTS this is 2.1.0-beta3) we should
> now be able to use 2GB correct?
>
> Thanks,
> Peter