You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by "Chou, Peter" <pb...@labs.att.com> on 2019/11/21 00:49:27 UTC
Question on calling TSMimeHdrFieldFind() from the Lua plugin.
Hi All,
We experienced some core dumps in ATS 7.1.4. My question is whether the pointers
returned by TSHttpTxnServerReqGet() might ever become invalid over the life of the transaction?
The core dump occurred from the Lua plugin in the ts_lua_server_request_header_set() function --
if (!http_ctx->server_request_hdrp) {
if (TSHttpTxnServerReqGet(http_ctx->txnp, &http_ctx->server_request_bufp, &http_ctx->server_request_hdrp) != TS_SUCCESS) {
return 0;
}
}
field_loc = TSMimeHdrFieldFind(http_ctx->server_request_bufp, http_ctx->server_request_hdrp, key, key_len);
--once inside TSMimeHdrFieldFind() one of the sanity checks fails which causes a core to be generated. Should the sanity
be pre-checked before we call out of the Lua plugin, or should we always execute the TSHttpTxnServerReqGet() call
even if pointers were non-null from a previous call?
Thanks,
Peter
#0 0x00002b46bdadb207 in raise () from /lib64/libc.so.6
#0 0x00002b46bdadb207 in raise () from /lib64/libc.so.6
#1 0x00002b46bdadca38 in abort () from /lib64/libc.so.6
#2 0x00002b46bb149eeb in ink_abort (message_format=message_format@entry=0x2b46bb156767 "%s:%d: failed assertion `%s`") at ink_error.cc:99
#3 0x00002b46bb147f75 in _ink_assert (expression=<optimized out>, file=<optimized out>, line=<optimized out>) at ink_assert.cc:37
#4 0x00000000004c9f16 in TSMimeHdrFieldFind ()
#5 0x00002b46c0789f80 in ts_lua_server_request_header_set (L=0x4090ea60) at experimental/ts_lua/ts_lua_server_request.c:240
#6 0x0000000000565a86 in lj_BC_FUNCC ()
#7 0x00002b46c078d5d2 in ts_lua_http_cont_handler (contp=<optimized out>, ev=TS_EVENT_HTTP_SEND_REQUEST_HDR, edata=0x4f8ad40)
at experimental/ts_lua/ts_lua_util.c:778
#8 0x00000000004c5f5e in INKContInternal::handle_event(int, void*) ()
#9 0x0000000000589c64 in HttpSM::state_api_callout(int, void*) ()
#10 0x000000000058cc70 in HttpSM::handle_http_server_open() ()
#11 0x000000000058ec28 in HttpSM::state_http_server_open(int, void*) ()
#12 0x000000000059160d in HttpSM::main_handler(int, void*) ()
#13 0x000000000072cef9 in UnixNetVConnection::connectUp(EThread*, int) ()
#14 0x000000000072a4e3 in UnixNetProcessor::connect_re_internal(Continuation*, sockaddr const*, NetVCOptions*) ()
#15 0x000000000058d4af in HttpSM::do_http_server_open(bool) ()
#16 0x0000000000592883 in HttpSM::set_next_state() ()
#17 0x000000000057fa52 in HttpSM::call_transact_and_set_next_state(void (*)(HttpTransact::State*)) ()
#18 0x0000000000591b6a in HttpSM::handle_api_return() ()
#19 0x0000000000589e6e in HttpSM::state_api_callout(int, void*) ()
#20 0x000000000059258b in HttpSM::set_next_state() ()
#21 0x000000000057fa52 in HttpSM::call_transact_and_set_next_state(void (*)(HttpTransact::State*)) ()
#22 0x0000000000590300 in HttpSM::state_hostdb_lookup(int, void*) ()
#23 0x000000000059160d in HttpSM::main_handler(int, void*) ()
#24 0x0000000000661f80 in reply_to_cont(Continuation*, HostDBInfo*, bool) ()
#25 0x000000000066647c in HostDBContinuation::dnsEvent(int, HostEnt*) ()
#26 0x0000000000679de8 in DNSEntry::postEvent(int, Event*) ()
#27 0x000000000067da38 in dns_result(DNSHandler*, DNSEntry*, HostEnt*, bool) ()
#28 0x000000000067e20f in DNSHandler::recv_dns(int, Event*) ()
#29 0x0000000000680034 in DNSHandler::mainEvent(int, Event*) ()
#30 0x0000000000755d52 in EThread::execute_regular() ()
#31 0x0000000000754a1a in spawn_thread_internal(void*) ()
#32 0x00002b46bce6ddd5 in start_thread () from /lib64/libpthread.so.0
#33 0x00002b46bdba3b3d in clone () from /lib64/libc.so.6
Re: Question on calling TSMimeHdrFieldFind() from the Lua plugin.
Posted by SUSAN HINRICHS <sh...@ieee.org>.
Sorry for the delay. We should probably set up an issue to coordinate the
issue here. A couple weeks back, I was seeing a few use-after free crash
in an ASAN build that claim a HttpHdr is being used from a Lua call and
that HttpHdr had been freed from another thread. Around the same time we
got a core dump like you saw. I was able to grub through the core and
determined that the cached hdrp value did not match the hdrp value being
used deeper in the stack.
Talking with Alan, we theorized that in some HdrHeap transformations the
m_http pointer may change. So we added logic to fetch the hdrp value each
time and assert if it did not match the previously fetched value.
Unfortunately, I haven't seen that assert twig yet. We are still seeing a
variety of crashes in the lua area, and sometimes lockups in the lua GC.
Trying to figure out what we changed in our environment around the time
that these GC lockups started.
On Wed, Nov 20, 2019 at 9:09 PM SUSAN HINRICHS <sh...@ieee.org> wrote:
> Yes, we have seen something similar to this in the last week or so. By
> the time we deployed a build that fetches the hdrp on each call and asserts
> if the differ, the crashes we had been seeing cleared up.
>
> I will review my notes on this when I get back to the office tomorrow.
>
> Susan
>
> On Wed, Nov 20, 2019, 6:50 PM Chou, Peter <pb...@labs.att.com> wrote:
>
>> Hi All,
>>
>> We experienced some core dumps in ATS 7.1.4. My question is whether the
>> pointers
>> returned by TSHttpTxnServerReqGet() might ever become invalid over the
>> life of the transaction?
>> The core dump occurred from the Lua plugin in the
>> ts_lua_server_request_header_set() function --
>>
>> if (!http_ctx->server_request_hdrp) {
>> if (TSHttpTxnServerReqGet(http_ctx->txnp,
>> &http_ctx->server_request_bufp, &http_ctx->server_request_hdrp) !=
>> TS_SUCCESS) {
>> return 0;
>> }
>> }
>>
>> field_loc = TSMimeHdrFieldFind(http_ctx->server_request_bufp,
>> http_ctx->server_request_hdrp, key, key_len);
>>
>> --once inside TSMimeHdrFieldFind() one of the sanity checks fails which
>> causes a core to be generated. Should the sanity
>> be pre-checked before we call out of the Lua plugin, or should we always
>> execute the TSHttpTxnServerReqGet() call
>> even if pointers were non-null from a previous call?
>>
>> Thanks,
>> Peter
>>
>> #0 0x00002b46bdadb207 in raise () from /lib64/libc.so.6
>> #0 0x00002b46bdadb207 in raise () from /lib64/libc.so.6
>> #1 0x00002b46bdadca38 in abort () from /lib64/libc.so.6
>> #2 0x00002b46bb149eeb in ink_abort (message_format=message_format@entry=0x2b46bb156767
>> "%s:%d: failed assertion `%s`") at ink_error.cc:99
>> #3 0x00002b46bb147f75 in _ink_assert (expression=<optimized out>,
>> file=<optimized out>, line=<optimized out>) at ink_assert.cc:37
>> #4 0x00000000004c9f16 in TSMimeHdrFieldFind ()
>> #5 0x00002b46c0789f80 in ts_lua_server_request_header_set (L=0x4090ea60)
>> at experimental/ts_lua/ts_lua_server_request.c:240
>> #6 0x0000000000565a86 in lj_BC_FUNCC ()
>> #7 0x00002b46c078d5d2 in ts_lua_http_cont_handler (contp=<optimized
>> out>, ev=TS_EVENT_HTTP_SEND_REQUEST_HDR, edata=0x4f8ad40)
>> at experimental/ts_lua/ts_lua_util.c:778
>> #8 0x00000000004c5f5e in INKContInternal::handle_event(int, void*) ()
>> #9 0x0000000000589c64 in HttpSM::state_api_callout(int, void*) ()
>> #10 0x000000000058cc70 in HttpSM::handle_http_server_open() ()
>> #11 0x000000000058ec28 in HttpSM::state_http_server_open(int, void*) ()
>> #12 0x000000000059160d in HttpSM::main_handler(int, void*) ()
>> #13 0x000000000072cef9 in UnixNetVConnection::connectUp(EThread*, int) ()
>> #14 0x000000000072a4e3 in
>> UnixNetProcessor::connect_re_internal(Continuation*, sockaddr const*,
>> NetVCOptions*) ()
>> #15 0x000000000058d4af in HttpSM::do_http_server_open(bool) ()
>> #16 0x0000000000592883 in HttpSM::set_next_state() ()
>> #17 0x000000000057fa52 in HttpSM::call_transact_and_set_next_state(void
>> (*)(HttpTransact::State*)) ()
>> #18 0x0000000000591b6a in HttpSM::handle_api_return() ()
>> #19 0x0000000000589e6e in HttpSM::state_api_callout(int, void*) ()
>> #20 0x000000000059258b in HttpSM::set_next_state() ()
>> #21 0x000000000057fa52 in HttpSM::call_transact_and_set_next_state(void
>> (*)(HttpTransact::State*)) ()
>> #22 0x0000000000590300 in HttpSM::state_hostdb_lookup(int, void*) ()
>> #23 0x000000000059160d in HttpSM::main_handler(int, void*) ()
>> #24 0x0000000000661f80 in reply_to_cont(Continuation*, HostDBInfo*, bool)
>> ()
>> #25 0x000000000066647c in HostDBContinuation::dnsEvent(int, HostEnt*) ()
>> #26 0x0000000000679de8 in DNSEntry::postEvent(int, Event*) ()
>> #27 0x000000000067da38 in dns_result(DNSHandler*, DNSEntry*, HostEnt*,
>> bool) ()
>> #28 0x000000000067e20f in DNSHandler::recv_dns(int, Event*) ()
>> #29 0x0000000000680034 in DNSHandler::mainEvent(int, Event*) ()
>> #30 0x0000000000755d52 in EThread::execute_regular() ()
>> #31 0x0000000000754a1a in spawn_thread_internal(void*) ()
>> #32 0x00002b46bce6ddd5 in start_thread () from /lib64/libpthread.so.0
>> #33 0x00002b46bdba3b3d in clone () from /lib64/libc.so.6
>>
>
Re: Question on calling TSMimeHdrFieldFind() from the Lua plugin.
Posted by SUSAN HINRICHS <sh...@ieee.org>.
Yes, we have seen something similar to this in the last week or so. By the
time we deployed a build that fetches the hdrp on each call and asserts if
the differ, the crashes we had been seeing cleared up.
I will review my notes on this when I get back to the office tomorrow.
Susan
On Wed, Nov 20, 2019, 6:50 PM Chou, Peter <pb...@labs.att.com> wrote:
> Hi All,
>
> We experienced some core dumps in ATS 7.1.4. My question is whether the
> pointers
> returned by TSHttpTxnServerReqGet() might ever become invalid over the
> life of the transaction?
> The core dump occurred from the Lua plugin in the
> ts_lua_server_request_header_set() function --
>
> if (!http_ctx->server_request_hdrp) {
> if (TSHttpTxnServerReqGet(http_ctx->txnp,
> &http_ctx->server_request_bufp, &http_ctx->server_request_hdrp) !=
> TS_SUCCESS) {
> return 0;
> }
> }
>
> field_loc = TSMimeHdrFieldFind(http_ctx->server_request_bufp,
> http_ctx->server_request_hdrp, key, key_len);
>
> --once inside TSMimeHdrFieldFind() one of the sanity checks fails which
> causes a core to be generated. Should the sanity
> be pre-checked before we call out of the Lua plugin, or should we always
> execute the TSHttpTxnServerReqGet() call
> even if pointers were non-null from a previous call?
>
> Thanks,
> Peter
>
> #0 0x00002b46bdadb207 in raise () from /lib64/libc.so.6
> #0 0x00002b46bdadb207 in raise () from /lib64/libc.so.6
> #1 0x00002b46bdadca38 in abort () from /lib64/libc.so.6
> #2 0x00002b46bb149eeb in ink_abort (message_format=message_format@entry=0x2b46bb156767
> "%s:%d: failed assertion `%s`") at ink_error.cc:99
> #3 0x00002b46bb147f75 in _ink_assert (expression=<optimized out>,
> file=<optimized out>, line=<optimized out>) at ink_assert.cc:37
> #4 0x00000000004c9f16 in TSMimeHdrFieldFind ()
> #5 0x00002b46c0789f80 in ts_lua_server_request_header_set (L=0x4090ea60)
> at experimental/ts_lua/ts_lua_server_request.c:240
> #6 0x0000000000565a86 in lj_BC_FUNCC ()
> #7 0x00002b46c078d5d2 in ts_lua_http_cont_handler (contp=<optimized out>,
> ev=TS_EVENT_HTTP_SEND_REQUEST_HDR, edata=0x4f8ad40)
> at experimental/ts_lua/ts_lua_util.c:778
> #8 0x00000000004c5f5e in INKContInternal::handle_event(int, void*) ()
> #9 0x0000000000589c64 in HttpSM::state_api_callout(int, void*) ()
> #10 0x000000000058cc70 in HttpSM::handle_http_server_open() ()
> #11 0x000000000058ec28 in HttpSM::state_http_server_open(int, void*) ()
> #12 0x000000000059160d in HttpSM::main_handler(int, void*) ()
> #13 0x000000000072cef9 in UnixNetVConnection::connectUp(EThread*, int) ()
> #14 0x000000000072a4e3 in
> UnixNetProcessor::connect_re_internal(Continuation*, sockaddr const*,
> NetVCOptions*) ()
> #15 0x000000000058d4af in HttpSM::do_http_server_open(bool) ()
> #16 0x0000000000592883 in HttpSM::set_next_state() ()
> #17 0x000000000057fa52 in HttpSM::call_transact_and_set_next_state(void
> (*)(HttpTransact::State*)) ()
> #18 0x0000000000591b6a in HttpSM::handle_api_return() ()
> #19 0x0000000000589e6e in HttpSM::state_api_callout(int, void*) ()
> #20 0x000000000059258b in HttpSM::set_next_state() ()
> #21 0x000000000057fa52 in HttpSM::call_transact_and_set_next_state(void
> (*)(HttpTransact::State*)) ()
> #22 0x0000000000590300 in HttpSM::state_hostdb_lookup(int, void*) ()
> #23 0x000000000059160d in HttpSM::main_handler(int, void*) ()
> #24 0x0000000000661f80 in reply_to_cont(Continuation*, HostDBInfo*, bool)
> ()
> #25 0x000000000066647c in HostDBContinuation::dnsEvent(int, HostEnt*) ()
> #26 0x0000000000679de8 in DNSEntry::postEvent(int, Event*) ()
> #27 0x000000000067da38 in dns_result(DNSHandler*, DNSEntry*, HostEnt*,
> bool) ()
> #28 0x000000000067e20f in DNSHandler::recv_dns(int, Event*) ()
> #29 0x0000000000680034 in DNSHandler::mainEvent(int, Event*) ()
> #30 0x0000000000755d52 in EThread::execute_regular() ()
> #31 0x0000000000754a1a in spawn_thread_internal(void*) ()
> #32 0x00002b46bce6ddd5 in start_thread () from /lib64/libpthread.so.0
> #33 0x00002b46bdba3b3d in clone () from /lib64/libc.so.6
>