You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by GitBox <gi...@apache.org> on 2022/01/29 01:06:29 UTC
[GitHub] [trafficserver] pbchou opened a new issue #8635: Core dump due to assertion failure "vio->mutex->thread_holding == this_ethread()" in PluginVC.cc with Lua plugin
pbchou opened a new issue #8635:
URL: https://github.com/apache/trafficserver/issues/8635
With an '--enable-debug' compiled version of ATS 9.1.1, we are seeing a core dump due to assertion failure "vio->mutex->thread_holding == this_ethread()" in PluginVC.cc with the Lua plugin. This also appears to happen with the ATS 9.1.x and ATS master branches. The situation is that we are running a custom Lua script that does token hash authorization. If the authorization fails, then we use ts.http.intercept() to send the client a 401 error response. Everything works correctly with the non-debug compiled version, so perhaps this assertion is not correct for this case?
Mention @oknet who added the assertion.
GDB back-trace follows --
```
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ffff7170859 in __GI_abort () at abort.c:79
#2 0x00007ffff7f1b67c in ink_abort (message_format=0x7ffff7f84ed0 "%s:%d: failed assertion `%s`") at ink_error.cc:99
#3 0x00007ffff7f182a9 in _ink_assert (expression=0x555555bf78f8 "vio->mutex->thread_holding == this_ethread()", file=0x555555bf7738 "PluginVC.cc", line=313) at ink_assert.cc:37
#4 0x0000555555ad6ff7 in PluginVC::reenable (this=0x7ffff09c1328, vio=0x7ffff09c1558) at PluginVC.cc:313
#5 0x0000555555816543 in VIO::reenable (this=0x7ffff09c1558) at /home/pebc/git/codecloud/trafficserver/iocore/eventsystem/P_VIO.h:100
#6 0x0000555555850e13 in TSVIOReenable (viop=0x7ffff09c1558) at traffic_server/InkIOCoreAPI.cc:330
#7 0x00007ffff0ce183b in ts_lua_say (L=0x7fffe4661040) at lua/ts_lua_http_intercept.c:369
#8 0x00007ffff0c50a46 in ?? () from /home/pebc/opt/LuaJIT/lib/libluajit-5.1.so.2
#9 0x00007ffff0ce1520 in ts_lua_http_intercept_run_coroutine (ictx=0x7ffff09c0000, n=0) at lua/ts_lua_http_intercept.c:257
#10 0x00007ffff0ce1302 in ts_lua_http_intercept_process (ictx=0x7ffff09c0000, conn=0x7ffff09c1328) at lua/ts_lua_http_intercept.c:191
#11 0x00007ffff0ce1217 in ts_lua_http_intercept_entry (contp=0x7ffff1c73cd0, event=TS_EVENT_NET_ACCEPT, edata=0x7ffff09c1328) at lua/ts_lua_http_intercept.c:147
#12 0x00005555558250a6 in INKContInternal::handle_event (this=0x7ffff1c73cd0, event=202, edata=0x7ffff09c1328) at traffic_server/InkAPI.cc:1137
#13 0x0000555555816315 in Continuation::handleEvent (this=0x7ffff1c73cd0, event=202, data=0x7ffff09c1328) at /home/pebc/git/codecloud/trafficserver/iocore/eventsystem/I_Continuation.h:219
#14 0x0000555555ada58e in PluginVCCore::state_send_accept (this=0x7ffff09c1000) at PluginVC.cc:1185
#15 0x0000555555ada273 in PluginVCCore::connect_re (this=0x7ffff09c1000, c=0x7fffe09a0ba0) at PluginVC.cc:1142
#16 0x00005555558cc79b in HttpSM::do_http_server_open (this=0x7fffe09a0ba0, raw=false) at HttpSM.cc:4956
#17 0x00005555558d86e1 in HttpSM::set_next_state (this=0x7fffe09a0ba0) at HttpSM.cc:7588
#18 0x00005555558d7945 in HttpSM::call_transact_and_set_next_state (this=0x7fffe09a0ba0, f=0x0) at HttpSM.cc:7397
#19 0x00005555558bd758 in HttpSM::handle_api_return (this=0x7fffe09a0ba0) at HttpSM.cc:1706
#20 0x00005555558bd418 in HttpSM::state_api_callout (this=0x7fffe09a0ba0, event=0, data=0x0) at HttpSM.cc:1638
#21 0x00005555558ce70c in HttpSM::do_api_callout_internal (this=0x7fffe09a0ba0) at HttpSM.cc:5360
#22 0x00005555558e191b in HttpSM::do_api_callout (this=0x7fffe09a0ba0) at HttpSM.cc:442
#23 0x00005555558d79ee in HttpSM::set_next_state (this=0x7fffe09a0ba0) at HttpSM.cc:7431
#24 0x00005555558d7945 in HttpSM::call_transact_and_set_next_state (this=0x7fffe09a0ba0, f=0x0) at HttpSM.cc:7397
#25 0x00005555558d7b6a in HttpSM::set_next_state (this=0x7fffe09a0ba0) at HttpSM.cc:7450
#26 0x00005555558d7945 in HttpSM::call_transact_and_set_next_state (this=0x7fffe09a0ba0, f=0x0) at HttpSM.cc:7397
#27 0x00005555558bd758 in HttpSM::handle_api_return (this=0x7fffe09a0ba0) at HttpSM.cc:1706
#28 0x00005555558e1934 in HttpSM::do_api_callout (this=0x7fffe09a0ba0) at HttpSM.cc:444
#29 0x00005555558d79ee in HttpSM::set_next_state (this=0x7fffe09a0ba0) at HttpSM.cc:7431
#30 0x00005555558d7945 in HttpSM::call_transact_and_set_next_state (this=0x7fffe09a0ba0, f=0x0) at HttpSM.cc:7397
#31 0x00005555558bd758 in HttpSM::handle_api_return (this=0x7fffe09a0ba0) at HttpSM.cc:1706
#32 0x00005555558e1934 in HttpSM::do_api_callout (this=0x7fffe09a0ba0) at HttpSM.cc:444
#33 0x00005555558d79ee in HttpSM::set_next_state (this=0x7fffe09a0ba0) at HttpSM.cc:7431
#34 0x00005555558d7945 in HttpSM::call_transact_and_set_next_state (this=0x7fffe09a0ba0, f=0x5555558f5782 <HttpTransact::ModifyRequest(HttpTransact::State*)>) at HttpSM.cc:7397
#35 0x00005555558ba48d in HttpSM::state_read_client_request_header (this=0x7fffe09a0ba0, event=100, data=0x7fffe0c1fdb8) at HttpSM.cc:930
#36 0x00005555558c2882 in HttpSM::main_handler (this=0x7fffe09a0ba0, event=100, data=0x7fffe0c1fdb8) at HttpSM.cc:2718
#37 0x0000555555816315 in Continuation::handleEvent (this=0x7fffe09a0ba0, event=100, data=0x7fffe0c1fdb8) at /home/pebc/git/codecloud/trafficserver/iocore/eventsystem/I_Continuation.h:219
#38 0x00005555558b8be4 in HttpSM::setup_client_read_request_header (this=0x7fffe09a0ba0) at HttpSM.cc:656
#39 0x00005555558bd6a9 in HttpSM::handle_api_return (this=0x7fffe09a0ba0) at HttpSM.cc:1684
#40 0x00005555558e1934 in HttpSM::do_api_callout (this=0x7fffe09a0ba0) at HttpSM.cc:444
#41 0x00005555558b7e1f in HttpSM::state_add_to_list (this=0x7fffe09a0ba0, event=0) at HttpSM.cc:471
#42 0x00005555558b8a45 in HttpSM::attach_client_session (this=0x7fffe09a0ba0, client_vc=0x7ffff09aeb68, buffer_reader=0x7ffff0900ea8) at HttpSM.cc:635
#43 0x0000555555adf080 in ProxyTransaction::new_transaction (this=0x7ffff09aeb68, from_early_data=false) at ProxyTransaction.cc:63
#44 0x0000555555890df4 in Http1ClientSession::new_transaction (this=0x7ffff09ae800) at Http1ClientSession.cc:453
#45 0x0000555555890aee in Http1ClientSession::release (this=0x7ffff09ae800, trans=0x7ffff09aeb68) at Http1ClientSession.cc:413
#46 0x0000555555891277 in Http1ClientSession::start (this=0x7ffff09ae800) at Http1ClientSession.cc:510
#47 0x0000555555add7b6 in ProxySession::handle_api_return (this=0x7ffff09ae800, event=60000) at ProxySession.cc:165
#48 0x0000555555add6c6 in ProxySession::do_api_callout (this=0x7ffff09ae800, id=TS_HTTP_SSN_START_HOOK) at ProxySession.cc:146
#49 0x000055555588fb9d in Http1ClientSession::new_connection (this=0x7ffff09ae800, new_vc=0x7fffe0c1fbd0, iobuf=0x7ffff0900e90, reader=0x7ffff0900ea8) at Http1ClientSession.cc:201
#50 0x0000555555885509 in HttpSessionAccept::accept (this=0x7ffff1fd9ac0, netvc=0x7fffe0c1fbd0, iobuf=0x7ffff0900e90, reader=0x7ffff0900ea8) at HttpSessionAccept.cc:59
#51 0x0000555555adc82a in ProtocolProbeTrampoline::ioCompletionEvent (this=0x7ffff08f6000, event=100, edata=0x7fffe0c1fdb8) at ProtocolProbeSessionAccept.cc:151
#52 0x0000555555816315 in Continuation::handleEvent (this=0x7ffff08f6000, event=100, data=0x7fffe0c1fdb8) at /home/pebc/git/codecloud/trafficserver/iocore/eventsystem/I_Continuation.h:219
#53 0x0000555555b3d005 in read_signal_and_update (event=100, vc=0x7fffe0c1fbd0) at UnixNetVConnection.cc:83
#54 0x0000555555b3dfd1 in read_from_net (nh=0x7ffff1ac4b50, vc=0x7fffe0c1fbd0, thread=0x7ffff1ac09c0) at UnixNetVConnection.cc:323
#55 0x0000555555b405f1 in UnixNetVConnection::net_read_io (this=0x7fffe0c1fbd0, nh=0x7ffff1ac4b50, lthread=0x7ffff1ac09c0) at UnixNetVConnection.cc:921
#56 0x0000555555b32d8c in NetHandler::process_ready_list (this=0x7ffff1ac4b50) at UnixNet.cc:415
#57 0x0000555555b333c0 in NetHandler::waitForActivity (this=0x7ffff1ac4b50, timeout=60000000) at UnixNet.cc:552
#58 0x0000555555b775bc in EThread::execute_regular (this=0x7ffff1ac09c0) at UnixEThread.cc:303
#59 0x0000555555b7780a in EThread::execute (this=0x7ffff1ac09c0) at UnixEThread.cc:364
#60 0x0000555555b75f24 in spawn_thread_internal (a=0x7ffff6d29000) at Thread.cc:92
#61 0x00007ffff769a609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#62 0x00007ffff726d293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] bryancall commented on issue #8635: Core dump due to assertion failure "vio->mutex->thread_holding == this_ethread()" in PluginVC.cc with Lua plugin
Posted by GitBox <gi...@apache.org>.
bryancall commented on issue #8635:
URL: https://github.com/apache/trafficserver/issues/8635#issuecomment-1026356644
What is the value of `vio->mutex->thread_holding`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] pbchou commented on issue #8635: Core dump due to assertion failure "vio->mutex->thread_holding == this_ethread()" in PluginVC.cc with Lua plugin
Posted by GitBox <gi...@apache.org>.
pbchou commented on issue #8635:
URL: https://github.com/apache/trafficserver/issues/8635#issuecomment-1068433242
FYI. I responded to PR #8687 that the fix seemed to solve my issue. I am OK to close this issue as long this reference PR is approved. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] pbchou commented on issue #8635: Core dump due to assertion failure "vio->mutex->thread_holding == this_ethread()" in PluginVC.cc with Lua plugin
Posted by GitBox <gi...@apache.org>.
pbchou commented on issue #8635:
URL: https://github.com/apache/trafficserver/issues/8635#issuecomment-1028476384
Confirmed same result on up to date branches '9.1.x' and 'master'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] pbchou commented on issue #8635: Core dump due to assertion failure "vio->mutex->thread_holding == this_ethread()" in PluginVC.cc with Lua plugin
Posted by GitBox <gi...@apache.org>.
pbchou commented on issue #8635:
URL: https://github.com/apache/trafficserver/issues/8635#issuecomment-1028287901
The value of ```vio->mutex->thread_holding``` is a null pointer 0x0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] oknet commented on issue #8635: Core dump due to assertion failure "vio->mutex->thread_holding == this_ethread()" in PluginVC.cc with Lua plugin
Posted by GitBox <gi...@apache.org>.
oknet commented on issue #8635:
URL: https://github.com/apache/trafficserver/issues/8635#issuecomment-1044583075
There are 2 TSConts and 2 TSMutex created:
- The first at `ts_lua_http_intercept` and `ts_lua_http_server_intercept`, and the handler is `ts_lua_http_intercept_entry`
https://github.com/apache/trafficserver/blob/eaf9848fd539d7f42218374b9f65754df943956e/plugins/lua/ts_lua_http_intercept.c#L86-L88
https://github.com/apache/trafficserver/blob/eaf9848fd539d7f42218374b9f65754df943956e/plugins/lua/ts_lua_http_intercept.c#L120-L122
- The second at `ts_lua_http_intercept_process`, and the handler is `ts_lua_http_intercept_handler`
https://github.com/apache/trafficserver/blob/eaf9848fd539d7f42218374b9f65754df943956e/plugins/lua/ts_lua_http_intercept.c#L170-L171
At the frame #11, `ts_lua` received `TS_EVENT_NET_ACCEPT` event and the first TSMutex is locked.
At the frame #7, `ts_lua` run into the `ts_lua_say` without `TS_EVENT_VCONN_WRITE_READY` event called back, it means the second TSMutex is not locked.
Although the current code does not present any problems, however, this is incorrect code.
To fix this issue:
- Suggestion: just reuse the first TSMutex in `ts_lua_http_intercept_process`,
- NOT RECOMMENDED: ignore `ink_assert` because there is a `SCOPED_MUTEX_LOCK` for VIO mutex after `ink_assert`.
The `ink_assert` statement is here to find this kind of problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] oknet commented on issue #8635: Core dump due to assertion failure "vio->mutex->thread_holding == this_ethread()" in PluginVC.cc with Lua plugin
Posted by GitBox <gi...@apache.org>.
oknet commented on issue #8635:
URL: https://github.com/apache/trafficserver/issues/8635#issuecomment-1046443297
@pbchou please test the PR #8687 , thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org