You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by "CHOU, PETER" <pc...@att.com> on 2022/06/28 21:16:06 UTC
RE: Lua plugin assertion failure question.
Kit,
This problem originally happened to us on 7.1.4 (see below), but it
seemed to stop after a while. I thought it might be fixed, but now it
has happened a few times with 9.1.2. It is very rare, happening three
times on two of our caches in the last month or so.
My question is this. Do you think it is sufficient to clear the saved
server_request_url and server_request_hdrp pointers in the
call-back for the TS_EVENT_HTTP_SEND_REQUEST_HDR hook similar to what is
done for the similar issue for the client response (7236e), or
would it be better to clear the pointers and do the TSHandleMLocRelease
within each function in ts_lua_server_request.c ( thus doing the
release within the same call-back ) ?
Thanks,
Peter
-----Original Message-----
From: Shu Kit Chan <ch...@gmail.com>
Sent: Monday, June 22, 2020 10:36 PM
To: dev <de...@trafficserver.apache.org>
Subject: Re: Lua plugin assertion failure question.
I actually haven't experienced this before for the server request
You mentioned in another thread that this may be related to you guys
setting open_write_fail_action to 4 in parent layer ?
If you can consistently reproduce this, you can perhaps add the
protection similar to client response and see if it is solving the
coredump problem.
I believe that it should not be too much of a problem in terms of efficiency.
Thanks.
Kit
On Wed, Jun 17, 2020 at 5:11 PM Chou, Peter <pb...@labs.att.com> wrote:
>
> Hi All,
>
> We recently experienced a bunch of assertion failures with the Lua plugin on ATS 7.1.4. When the transaction is being killed, the Lua plugin calls ts_lua_destroy_http_ctx() which calls TSHandleMLocRelease() with http_ctx->server_request_bufp as the buffer argument. The assertion failure is -
> Fatal: InkAPI.cc:1917: failed assertion `sdk_sanity_check_mbuffer(bufp) == TS_SUCCESS`
>
> Is it possible that the data structure for the server request is changed within the life-time of the transaction making the pointer invalid? I noticed that this was possible
> for client responses addressed in 7236e excerpt -
>
> + // client response can be changed within a transaction
> + // (e.g. due to the follow redirect feature). So, clearing the pointers
> + // to allow API(s) to fetch the pointers again when it re-enters the hook
> + if (http_ctx->client_response_hdrp != NULL) {
> + TSHandleMLocRelease(http_ctx->client_response_bufp, TS_NULL_MLOC, http_ctx->client_response_hdrp);
> + http_ctx->client_response_bufp = NULL;
> + http_ctx->client_response_hdrp = NULL;
> + }
>
> Is something similar possible with the server request which uses the same kind of null pointer check around the setting of the bufp and hdrp pointers?
>
> Back-trace -
>
> (gdb) bt
> #0 0x00002b94af14f377 in raise () from /lib64/libc.so.6
> #1 0x00002b94af150ba8 in abort () from /lib64/libc.so.6
> #2 0x00002b94ac7bdeeb in ink_abort (message_format=message_format@entry=0x2b94ac7ca767 "%s:%d: failed assertion `%s`")
> at ink_error.cc:99
> #3 0x00002b94ac7bbf75 in _ink_assert (expression=<optimized out>, file=<optimized out>, line=<optimized out>)
> at ink_assert.cc:37
> #4 0x00000000004c827a in TSHandleMLocRelease ()
> #5 0x00002b94b3c51f08 in ts_lua_destroy_http_ctx (http_ctx=0x2b94e7d85a20) at experimental/ts_lua/ts_lua_util.c:464
> #6 0x00002b94b3c5271a in ts_lua_http_cont_handler (contp=<optimized out>, ev=TS_EVENT_HTTP_TXN_CLOSE, edata=0x2b94b4d39340)
> at experimental/ts_lua/ts_lua_util.c:791
> #7 0x00000000004c5fde in INKContInternal::handle_event(int, void*) ()
> #8 0x0000000000589ce4 in HttpSM::state_api_callout(int, void*) ()
> #9 0x0000000000590a7d in HttpSM::kill_this() ()
> #10 0x00000000005916d8 in HttpSM::main_handler(int, void*) ()
> #11 0x000000000072f103 in UnixNetVConnection::mainEvent(int, Event*) ()
> #12 0x0000000000726459 in InactivityCop::check_inactivity(int, Event*) ()
> #13 0x0000000000755040 in EThread::process_event(Event*, int) ()
> #14 0x0000000000755da1 in EThread::execute_regular() ()
> #15 0x0000000000754a9a in spawn_thread_internal(void*) ()
> #16 0x00002b94ae4e1ea5 in start_thread () from /lib64/libpthread.so.0
> #17 0x00002b94af2178cd in clone () from /lib64/libc.so.6
>
> Thanks,
> Peter
RE: Lua plugin assertion failure question.
Posted by "CHOU, PETER" <pc...@att.com>.
Kit,
We will try the patch and see if the problem still occurs.
Thanks,
Peter
-----Original Message-----
From: Shu Kit Chan <ch...@gmail.com>
Sent: Tuesday, June 28, 2022 4:42 PM
To: dev <de...@trafficserver.apache.org>
Subject: Re: Lua plugin assertion failure question.
You can check out https://urldefense.com/v3/__https://github.com/apache/trafficserver/pull/8607__;!!BhdT!lFEKdtxbGBnhcNPCzFPi2u3EzgDOmNN3xGseGw7u7EP6beRGjrNIyDInP4VMtptmCMGtD5cDzstdxXXjFA$
That should essentially be doing what you want for server request pointers as well.
Can you try this out?
On Tue, Jun 28, 2022 at 2:16 PM CHOU, PETER <pc...@att.com> wrote:
>
> Kit,
>
> This problem originally happened to us on 7.1.4 (see below), but it
> seemed to stop after a while. I thought it might be fixed, but now it
> has happened a few times with 9.1.2. It is very rare, happening three
> times on two of our caches in the last month or so.
>
> My question is this. Do you think it is sufficient to clear the saved
> server_request_url and server_request_hdrp pointers in the call-back
> for the TS_EVENT_HTTP_SEND_REQUEST_HDR hook similar to what is done
> for the similar issue for the client response (7236e), or would it be
> better to clear the pointers and do the TSHandleMLocRelease within
> each function in ts_lua_server_request.c ( thus doing the release
> within the same call-back ) ?
>
> Thanks,
> Peter
>
> -----Original Message-----
> From: Shu Kit Chan <ch...@gmail.com>
> Sent: Monday, June 22, 2020 10:36 PM
> To: dev <de...@trafficserver.apache.org>
> Subject: Re: Lua plugin assertion failure question.
>
> I actually haven't experienced this before for the server request You
> mentioned in another thread that this may be related to you guys
> setting open_write_fail_action to 4 in parent layer ?
>
> If you can consistently reproduce this, you can perhaps add the
> protection similar to client response and see if it is solving the
> coredump problem.
> I believe that it should not be too much of a problem in terms of efficiency.
>
> Thanks.
>
> Kit
>
> On Wed, Jun 17, 2020 at 5:11 PM Chou, Peter <pb...@labs.att.com> wrote:
> >
> > Hi All,
> >
> > We recently experienced a bunch of assertion failures with the Lua
> > plugin on ATS 7.1.4. When the transaction is being killed, the Lua
> > plugin calls ts_lua_destroy_http_ctx() which calls
> > TSHandleMLocRelease() with http_ctx->server_request_bufp as the
> > buffer argument. The assertion failure is -
> > Fatal: InkAPI.cc:1917: failed assertion
> > `sdk_sanity_check_mbuffer(bufp) == TS_SUCCESS`
> >
> > Is it possible that the data structure for the server request is
> > changed within the life-time of the transaction making the pointer
> > invalid? I noticed that this was possible for client responses
> > addressed in 7236e excerpt -
> >
> > + // client response can be changed within a transaction // (e.g.
> > + due to the follow redirect feature). So, clearing the pointers //
> > + to allow API(s) to fetch the pointers again when it re-enters the
> > + hook if (http_ctx->client_response_hdrp != NULL) {
> > + TSHandleMLocRelease(http_ctx->client_response_bufp, TS_NULL_MLOC, http_ctx->client_response_hdrp);
> > + http_ctx->client_response_bufp = NULL;
> > + http_ctx->client_response_hdrp = NULL; }
> >
> > Is something similar possible with the server request which uses the same kind of null pointer check around the setting of the bufp and hdrp pointers?
> >
> > Back-trace -
> >
> > (gdb) bt
> > #0 0x00002b94af14f377 in raise () from /lib64/libc.so.6
> > #1 0x00002b94af150ba8 in abort () from /lib64/libc.so.6
> > #2 0x00002b94ac7bdeeb in ink_abort (message_format=message_format@entry=0x2b94ac7ca767 "%s:%d: failed assertion `%s`")
> > at ink_error.cc:99
> > #3 0x00002b94ac7bbf75 in _ink_assert (expression=<optimized out>, file=<optimized out>, line=<optimized out>)
> > at ink_assert.cc:37
> > #4 0x00000000004c827a in TSHandleMLocRelease ()
> > #5 0x00002b94b3c51f08 in ts_lua_destroy_http_ctx
> > (http_ctx=0x2b94e7d85a20) at experimental/ts_lua/ts_lua_util.c:464
> > #6 0x00002b94b3c5271a in ts_lua_http_cont_handler (contp=<optimized out>, ev=TS_EVENT_HTTP_TXN_CLOSE, edata=0x2b94b4d39340)
> > at experimental/ts_lua/ts_lua_util.c:791
> > #7 0x00000000004c5fde in INKContInternal::handle_event(int, void*)
> > ()
> > #8 0x0000000000589ce4 in HttpSM::state_api_callout(int, void*) ()
> > #9 0x0000000000590a7d in HttpSM::kill_this() ()
> > #10 0x00000000005916d8 in HttpSM::main_handler(int, void*) ()
> > #11 0x000000000072f103 in UnixNetVConnection::mainEvent(int, Event*)
> > ()
> > #12 0x0000000000726459 in InactivityCop::check_inactivity(int,
> > Event*) ()
> > #13 0x0000000000755040 in EThread::process_event(Event*, int) ()
> > #14 0x0000000000755da1 in EThread::execute_regular() ()
> > #15 0x0000000000754a9a in spawn_thread_internal(void*) ()
> > #16 0x00002b94ae4e1ea5 in start_thread () from
> > /lib64/libpthread.so.0
> > #17 0x00002b94af2178cd in clone () from /lib64/libc.so.6
> >
> > Thanks,
> > Peter
Re: Lua plugin assertion failure question.
Posted by Shu Kit Chan <ch...@gmail.com>.
You can check out https://github.com/apache/trafficserver/pull/8607
That should essentially be doing what you want for server request
pointers as well.
Can you try this out?
On Tue, Jun 28, 2022 at 2:16 PM CHOU, PETER <pc...@att.com> wrote:
>
> Kit,
>
> This problem originally happened to us on 7.1.4 (see below), but it
> seemed to stop after a while. I thought it might be fixed, but now it
> has happened a few times with 9.1.2. It is very rare, happening three
> times on two of our caches in the last month or so.
>
> My question is this. Do you think it is sufficient to clear the saved
> server_request_url and server_request_hdrp pointers in the
> call-back for the TS_EVENT_HTTP_SEND_REQUEST_HDR hook similar to what is
> done for the similar issue for the client response (7236e), or
> would it be better to clear the pointers and do the TSHandleMLocRelease
> within each function in ts_lua_server_request.c ( thus doing the
> release within the same call-back ) ?
>
> Thanks,
> Peter
>
> -----Original Message-----
> From: Shu Kit Chan <ch...@gmail.com>
> Sent: Monday, June 22, 2020 10:36 PM
> To: dev <de...@trafficserver.apache.org>
> Subject: Re: Lua plugin assertion failure question.
>
> I actually haven't experienced this before for the server request
> You mentioned in another thread that this may be related to you guys
> setting open_write_fail_action to 4 in parent layer ?
>
> If you can consistently reproduce this, you can perhaps add the
> protection similar to client response and see if it is solving the
> coredump problem.
> I believe that it should not be too much of a problem in terms of efficiency.
>
> Thanks.
>
> Kit
>
> On Wed, Jun 17, 2020 at 5:11 PM Chou, Peter <pb...@labs.att.com> wrote:
> >
> > Hi All,
> >
> > We recently experienced a bunch of assertion failures with the Lua plugin on ATS 7.1.4. When the transaction is being killed, the Lua plugin calls ts_lua_destroy_http_ctx() which calls TSHandleMLocRelease() with http_ctx->server_request_bufp as the buffer argument. The assertion failure is -
> > Fatal: InkAPI.cc:1917: failed assertion `sdk_sanity_check_mbuffer(bufp) == TS_SUCCESS`
> >
> > Is it possible that the data structure for the server request is changed within the life-time of the transaction making the pointer invalid? I noticed that this was possible
> > for client responses addressed in 7236e excerpt -
> >
> > + // client response can be changed within a transaction
> > + // (e.g. due to the follow redirect feature). So, clearing the pointers
> > + // to allow API(s) to fetch the pointers again when it re-enters the hook
> > + if (http_ctx->client_response_hdrp != NULL) {
> > + TSHandleMLocRelease(http_ctx->client_response_bufp, TS_NULL_MLOC, http_ctx->client_response_hdrp);
> > + http_ctx->client_response_bufp = NULL;
> > + http_ctx->client_response_hdrp = NULL;
> > + }
> >
> > Is something similar possible with the server request which uses the same kind of null pointer check around the setting of the bufp and hdrp pointers?
> >
> > Back-trace -
> >
> > (gdb) bt
> > #0 0x00002b94af14f377 in raise () from /lib64/libc.so.6
> > #1 0x00002b94af150ba8 in abort () from /lib64/libc.so.6
> > #2 0x00002b94ac7bdeeb in ink_abort (message_format=message_format@entry=0x2b94ac7ca767 "%s:%d: failed assertion `%s`")
> > at ink_error.cc:99
> > #3 0x00002b94ac7bbf75 in _ink_assert (expression=<optimized out>, file=<optimized out>, line=<optimized out>)
> > at ink_assert.cc:37
> > #4 0x00000000004c827a in TSHandleMLocRelease ()
> > #5 0x00002b94b3c51f08 in ts_lua_destroy_http_ctx (http_ctx=0x2b94e7d85a20) at experimental/ts_lua/ts_lua_util.c:464
> > #6 0x00002b94b3c5271a in ts_lua_http_cont_handler (contp=<optimized out>, ev=TS_EVENT_HTTP_TXN_CLOSE, edata=0x2b94b4d39340)
> > at experimental/ts_lua/ts_lua_util.c:791
> > #7 0x00000000004c5fde in INKContInternal::handle_event(int, void*) ()
> > #8 0x0000000000589ce4 in HttpSM::state_api_callout(int, void*) ()
> > #9 0x0000000000590a7d in HttpSM::kill_this() ()
> > #10 0x00000000005916d8 in HttpSM::main_handler(int, void*) ()
> > #11 0x000000000072f103 in UnixNetVConnection::mainEvent(int, Event*) ()
> > #12 0x0000000000726459 in InactivityCop::check_inactivity(int, Event*) ()
> > #13 0x0000000000755040 in EThread::process_event(Event*, int) ()
> > #14 0x0000000000755da1 in EThread::execute_regular() ()
> > #15 0x0000000000754a9a in spawn_thread_internal(void*) ()
> > #16 0x00002b94ae4e1ea5 in start_thread () from /lib64/libpthread.so.0
> > #17 0x00002b94af2178cd in clone () from /lib64/libc.so.6
> >
> > Thanks,
> > Peter