You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by GitBox <gi...@apache.org> on 2020/11/19 18:03:11 UTC
[GitHub] [trafficserver] shinrich opened a new issue #7333: Crashes with 9.0.x
shinrich opened a new issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333
We resynced with the ASF 9.0.x branch as of 11/10. The resulting build crashes for us with production traffic within 15-60 minutes.
I tracked down the crashes to two commits. PR #7237 and PR #7278. Either one of these commits in the build results in a crash.
The crash with PR #7237 has the InactivityCop on the stack. It appears to be trying to send the inactivity timeout to a freed continuation that is still being referenced by the read.vio. I believe that I have a fix for this issue, and will post a PR shortly.
The crash associated with PR #7278 appears to be a more general use after free. We are setting up our prod sim environment with an ASAN build to try and catch a more useful error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7333: Crashes with 9.0.x
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333#issuecomment-730663513
Got an ASAN use-after free when running in our prod-sim environment
```
NOTE: using command line path as RUNROOT
Traffic Server 9.0.0.89 Nov 19 2020 21:01:20 atslab02.champaign.office.oath
traffic_server: using root directory '/home/y'
[2020-11-19 21:16:46][INFO ][_vcs/iplookup_cpp/src/IpLookup.cpp][init:88] Loading /home/y/libdata/geotargeting/datapacks/coregeo_ipdatapack_country
[2020-11-19 21:16:46][INFO ][_vcs/iplookup_cpp/src/IpLookup.cpp][init:124] Memory mapping - No preload./home/y/libdata/geotargeting/datapacks/coregeo_ipdatapack_country
[2020-11-19 21:16:46][INFO ][_vcs/iplookup_cpp/src/IpLookup.cpp][init:88] Loading /home/y/libdata/geotargeting/datapacks/coregeo_ipdatapack_country_ipv6
[2020-11-19 21:16:46][INFO ][_vcs/iplookup_cpp/src/IpLookup.cpp][init:124] Memory mapping - No preload./home/y/libdata/geotargeting/datapacks/coregeo_ipdatapack_country_ipv6
=================================================================
==10253==ERROR: AddressSanitizer: heap-use-after-free on address 0x61a00008bad0 at pc 0x000000ed60a8 bp 0x2ba09f4e2a40 sp 0x2ba09f4e2a30
WRITE of size 8 at 0x61a00008bad0 thread T20 ([ET_NET 18])
#0 0xed60a7 in UnixNetVConnection::do_io_close(int) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:677
#1 0xe3cee5 in SSLNetVConnection::do_io_close(int) ../../../../../../_vcs/trafficserver9/iocore/net/SSLNetVConnection.cc:918
#2 0xef03bc in UnixNetVConnection::migrateToCurrentThread(Continuation*, EThread*) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:1470
#3 0x873f98 in HttpSessionManager::acquire_session(sockaddr const*, ats::CryptoHash const&, HttpSM*, TSServerSessionSharingMatchMask, TSServerSessionSharingPoolType) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSessionManager.cc:418
#4 0x875457 in HttpSessionManager::acquire_session(Continuation*, sockaddr const*, char const*, ProxyTransaction*, HttpSM*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSessionManager.cc:381
#5 0x843e6d in HttpSM::do_http_server_open(bool) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:4973
#6 0x84f257 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7523
#7 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#8 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#9 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#10 0x2ba0a1669a5c in cb _vcs/quick_filter-9/quick_filter/quick_filter.cc:637
#11 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#12 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#13 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#14 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#15 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#16 0x84f95a in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7360
#17 0x8025a9 in HttpSM::do_hostdb_lookup() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:4180
#18 0x84f518 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7676
#19 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#20 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#21 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#22 0x2ba0a947bf6c in main_handler ../../../../../_vcs/trafficserver9/plugins/regex_revalidate/regex_revalidate.c:453
#23 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#24 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#25 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#26 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#27 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#28 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#29 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#30 0x2ba0a87d41cc in carpLookup(tsapi_cont*, TSEvent, void*) _vcs/carp-9/carp/carp.cc:767
#31 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#32 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#33 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#34 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#35 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#36 0x82d21f in HttpSM::setup_cache_lookup_complete_api() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2516
#37 0x82d21f in HttpSM::state_cache_open_read(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2578
#38 0x82f3dc in HttpSM::main_handler(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2620
#39 0x9506fc in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#40 0x9506fc in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#41 0x9506fc in HttpCacheSM::state_cache_open_read(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:133
#42 0xca38e1 in Cache::open_read(Continuation*, ats::CryptoHash const*, HTTPHdr*, OverridableHttpConfigParams const*, CacheFragType, char const*, int) ../../../../../../_vcs/trafficserver9/iocore/cache/CacheRead.cc:149
#43 0xc203ab in CacheProcessor::open_read(Continuation*, HttpCacheKey const*, HTTPHdr*, OverridableHttpConfigParams const*, long, CacheFragType) ../../../../../../_vcs/trafficserver9/iocore/cache/Cache.cc:3242
#44 0x94fe0b in HttpCacheSM::do_cache_open_read(HttpCacheKey const&) ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:273
#45 0x951186 in HttpCacheSM::open_read(HttpCacheKey const*, URL*, HTTPHdr*, OverridableHttpConfigParams const*, long) ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:307
#46 0x7f0f10 in HttpSM::do_cache_lookup_and_read() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:4611
#47 0x84ebd8 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7491
#48 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#49 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#50 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#51 0x2ba0a1669a5c in cb _vcs/quick_filter-9/quick_filter/quick_filter.cc:637
#52 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#53 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#54 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#55 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#56 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#57 0x84f95a in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7360
#58 0x84e631 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7742
#59 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#60 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#61 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#62 0x2ba0a6e39bb4 in http_hook _vcs/vz_connection-9/INKPluginInit.cc:427
#63 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#64 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#65 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#66 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#67 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#68 0x84f95a in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7360
#69 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#70 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#71 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#72 0x2ba0a8f54ae1 in health_check_origin ../../../../../_vcs/trafficserver9/plugins/healthchecks/healthchecks.c:541
#73 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#74 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#75 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#76 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#77 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#78 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#79 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#80 0x2ba0a8cf7398 in cont_rewrite_headers ../../../../../_vcs/trafficserver9/plugins/header_rewrite/header_rewrite.cc:308
#81 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#82 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#83 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#84 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#85 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#86 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#87 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#88 0x2ba0a87d633c in onReadRequest(tsapi_cont*, TSEvent, void*) _vcs/carp-9/carp/carp.cc:375
#89 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#90 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#91 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#92 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#93 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#94 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#95 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#96 0x2ba0a6e39bb4 in http_hook _vcs/vz_connection-9/INKPluginInit.cc:427
#97 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#98 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#99 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#100 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#101 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#102 0x84f95a in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7360
#103 0x816c91 in HttpSM::state_read_client_request_header(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:837
#104 0x82f5be in HttpSM::main_handler(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2617
#105 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#106 0x830b0f in HttpSM::do_api_callout() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:353
#107 0x830b0f in HttpSM::state_add_to_list(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:382
#108 0x836f2d in HttpSM::attach_client_session(ProxyTransaction*, IOBufferReader*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:558
#109 0x79c007 in Http1ClientSession::state_keep_alive(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:369
#110 0xed76df in read_signal_and_update ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:83
#111 0xe45d14 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) ../../../../../../_vcs/trafficserver9/iocore/net/SSLNetVConnection.cc:670
#112 0xeb0194 in NetHandler::process_ready_list() ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:413
#113 0xeb0f42 in NetHandler::waitForActivity(long) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:548
#114 0xfc80fe in EThread::execute_regular() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:271
#115 0xfc8955 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#116 0xfc8955 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#117 0xfc24ea in spawn_thread_internal ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
#118 0x2ba0948fcdd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
#119 0x2ba09583a02c in __clone (/lib64/libc.so.6+0xfe02c)
0x61a00008bad0 is located 592 bytes inside of 1264-byte region [0x61a00008b880,0x61a00008bd70)
freed by thread T11 ([ET_NET 9]) here:
#0 0x63cd20 in __interceptor_free (/opt/oath/trafficserver/9.0/bin/traffic_server+0x63cd20)
#1 0x2ba092f2007c in ink_freelist_free ../../../../../../_vcs/trafficserver9/src/tscore/ink_queue.cc:281
#2 0xeaf762 in NetHandler::free_netevent(NetEvent*) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:365
#3 0xeb01ba in NetHandler::process_ready_list() ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:428
#4 0xeb0f42 in NetHandler::waitForActivity(long) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:548
#5 0xfc80fe in EThread::execute_regular() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:271
#6 0xfc8955 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#7 0xfc8955 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#8 0xfc24ea in spawn_thread_internal ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
#9 0x2ba0948fcdd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
previously allocated by thread T11 ([ET_NET 9]) here:
#0 0x63df00 in __interceptor_posix_memalign (/opt/oath/trafficserver/9.0/bin/traffic_server+0x63df00)
#1 0x2ba092f1e10f in ats_memalign ../../../../../../_vcs/trafficserver9/src/tscore/ink_memory.cc:102
#2 0x2ba092f1ffde in ink_freelist_new ../../../../../../_vcs/trafficserver9/src/tscore/ink_queue.cc:187
#3 0xe2d900 in ClassAllocator<SSLNetVConnection>::alloc() /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/include/tscore/Allocator.h:131
#4 0xe2d900 in SSLNetVConnection* thread_alloc_init<SSLNetVConnection>(ClassAllocator<SSLNetVConnection>&, ProxyAllocator&) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_ProxyAllocator.h:73
#5 0xe2d900 in SSLNetProcessor::allocate_vc(EThread*) ../../../../../../_vcs/trafficserver9/iocore/net/SSLNetProcessor.cc:107
#6 0xef02ec in UnixNetVConnection::migrateToCurrentThread(Continuation*, EThread*) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:1448
#7 0x873f98 in HttpSessionManager::acquire_session(sockaddr const*, ats::CryptoHash const&, HttpSM*, TSServerSessionSharingMatchMask, TSServerSessionSharingPoolType) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSessionManager.cc:418
#8 0x875457 in HttpSessionManager::acquire_session(Continuation*, sockaddr const*, char const*, ProxyTransaction*, HttpSM*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSessionManager.cc:381
#9 0x843e6d in HttpSM::do_http_server_open(bool) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:4973
#10 0x84f257 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7523
#11 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#12 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#13 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#14 0x2ba0a1669a5c in cb _vcs/quick_filter-9/quick_filter/quick_filter.cc:637
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv edited a comment on issue #7333: Crashes with 9.0.x
Posted by GitBox <gi...@apache.org>.
sudheerv edited a comment on issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333#issuecomment-730636653
Ah, we have #7278 for about 2 months in our prod but are not seeing this possibly because we don't have this in our HTTP/2 env.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv commented on issue #7333: Crashes with 9.0.x
Posted by GitBox <gi...@apache.org>.
sudheerv commented on issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333#issuecomment-730636653
Ah, we are not seeing this because we don't have this in our HTTP/2 env.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich closed issue #7333: 9.0.x use-after-free crash
Posted by GitBox <gi...@apache.org>.
shinrich closed issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv commented on issue #7333: Crashes with 9.0.x
Posted by GitBox <gi...@apache.org>.
sudheerv commented on issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333#issuecomment-730723282
> T11 will be in the event loop looking at the close flag on the netvc. It will not try to acquire the read vio's mutex before calling close_vc.
Ah gotcha! As discussed on slack though, if we do not set `close` flag asap (or at least, prior to setting read.vio.cont to nullptr), a rescheduled read may end up getting unblocked (due to read.vio.cont.mutex not valid anymore) and perform net_read_io and can cause issues. One potential workaround might be to move setting close to latter, but, defer setting read.vio.cont and write.vio.cont to nullptr until `free()` is called.
I'm also not totally sure if this explains the h2 stack traces though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7333: Crashes with 9.0.x
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333#issuecomment-730721901
T11 will be in the event loop looking at the close flag on the netvc. It will not try to acquire the read vio's mutex before calling close_vc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7333: Crashes with 9.0.x
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333#issuecomment-730565362
Interestingly, the crashes associated with PR #7278 all seem to be with the Http2 client. Here are a couple stack traces.
```
(gdb) bt
#0 0x000000000061968f in get_header_field (index=<optimized out>, this=0x2b3b8eb3d3c0) at ../../../../../../_vcs/trafficserver9/proxy/http2/HPACK.cc:357
#1 get_header_field (field=..., index=<optimized out>, this=<optimized out>) at ../../../../../../_vcs/trafficserver9/proxy/http2/HPACK.cc:294
#2 HpackIndexingTable::get_header_field (this=<optimized out>, index=<optimized out>, field=...) at ../../../../../../_vcs/trafficserver9/proxy/http2/HPACK.cc:281
#3 0x0000000000619ed8 in decode_indexed_header_field(MIMEFieldWrapper&, unsigned char const*, unsigned char const*, HpackIndexingTable&) ()
at ../../../../../../_vcs/trafficserver9/proxy/http2/HPACK.cc:649
#4 0x000000000061b20f in hpack_decode_header_block (indexing_table=..., hdr=hdr@entry=0x2b3da9b40f58,
in_buf=0x2b3e97a31900 "\202\005\377\210\004b\277\361\060\001w\032u\241>\a\300\333N:\353A\370\213\214\034~\021", in_buf_len=<optimized out>, max_header_size=131072,
maximum_table_size=maximum_table_size@entry=4096) at ../../../../../../_vcs/trafficserver9/proxy/http2/HPACK.cc:812
#5 0x00000000005fb0bb in http2_decode_header_blocks (hdr=hdr@entry=0x2b3da9b40f58, buf_start=<optimized out>, buf_len=<optimized out>, len_read=len_read@entry=0x0, handle=...,
trailing_header=@0x2b3da9b41978: false, maximum_table_size=4096) at ../../../../../../_vcs/trafficserver9/proxy/http2/HTTP2.cc:690
#6 0x000000000060ea01 in Http2Stream::decode_header_blocks (this=this@entry=0x2b3da9b40e00, hpack_handle=..., maximum_table_size=<optimized out>)
at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2Stream.cc:158
#7 0x000000000060a6a2 in rcv_headers_frame(Http2ConnectionState&, Http2Frame const&) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ConnectionState.h:103
#8 0x000000000060c1a4 in Http2ConnectionState::main_event_handler(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ConnectionState.cc:1063
#9 0x00000000005ffac9 in handleEvent (data=0x2b3935901370, event=2253, this=0x2b3e97a82700)
at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#10 handleEvent (data=0x2b3935901370, event=2253, this=0x2b3e97a82700)
at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#11 send_connection_event(Continuation*, int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:65
#12 0x00000000006004f7 in Http2ClientSession::do_complete_frame_read (this=this@entry=0x2b3e97a82400) at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:568
#13 0x0000000000600985 in Http2ClientSession::state_process_frame_read(int, VIO*, bool) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:626
#14 0x00000000005ffdeb in Http2ClientSession::main_event_handler(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:352
#15 0x000000000076327e in read_signal_and_update(int, UnixNetVConnection*) () at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:83
#16 0x0000000000739a69 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) () at ../../../../../../_vcs/trafficserver9/iocore/net/SSLNetVConnection.cc:670
#17 0x0000000000757abe in NetHandler::process_ready_list (this=this@entry=0x2b3931736ae0) at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:413
#18 0x0000000000757f00 in NetHandler::waitForActivity(long) () at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:548
#19 0x00000000007a51e1 in EThread::execute_regular (this=this@entry=0x2b3931732a40) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_PriorityEventQueue.h:115
#20 0x00000000007a5436 in execute (this=0x2b3931732a40) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#21 EThread::execute (this=0x2b3931732a40) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#22 0x00000000007a3669 in spawn_thread_internal (a=0x2b392de7f100) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
#23 0x00002b392c2e9ea5 in start_thread () from /lib64/libpthread.so.0
#24 0x00002b392d01f8dd in clone () from /lib64/libc.so.6
```
```
(gdb) bt
#0 mime_hdr_destroy_field_block_list (head=0x594d2d31746c4867, heap=0x2ab7496c8000) at ../../../../../../_vcs/trafficserver9/proxy/hdrs/MIME.cc:1091
#1 mime_hdr_fields_clear (heap=0x2ab7496c8000, mh=0x2ab7496c8088) at ../../../../../../_vcs/trafficserver9/proxy/hdrs/MIME.cc:1232
#2 0x0000000000619914 in fields_clear (this=<optimized out>, this=<optimized out>)
at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/proxy/hdrs/MIME.h:1159
#3 HpackDynamicTable::~HpackDynamicTable (this=0x2ab81c5b2cf0, __in_chrg=<optimized out>) at ../../../../../../_vcs/trafficserver9/proxy/http2/HPACK.cc:343
#4 0x00000000006038a8 in ~HpackIndexingTable (this=0x2ab6f56c27e0, __in_chrg=<optimized out>) at ../../../../../../_vcs/trafficserver9/proxy/http2/HPACK.h:147
#5 Http2ConnectionState::destroy (this=this@entry=0x2ab830ab5700) at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ConnectionState.h:157
#6 0x00000000006020eb in Http2ClientSession::free() () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:160
#7 0x0000000000600021 in Http2ClientSession::main_event_handler(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:420
#8 0x000000000076327e in read_signal_and_update(int, UnixNetVConnection*) () at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:83
#9 0x000000000076752c in UnixNetVConnection::mainEvent(int, Event*) () at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:1189
#10 0x000000000075a445 in InactivityCop::check_inactivity (this=0x2ab4438cb000, event=<optimized out>, e=<optimized out>) at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:93
#11 0x00000000007a4113 in handleEvent (data=0x2ab443800060, event=2, this=0x2ab4438cb000) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#12 handleEvent (data=0x2ab443800060, event=2, this=0x2ab4438cb000) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#13 EThread::process_event(Event*, int) () at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:132
#14 0x00000000007a5024 in EThread::execute_regular (this=this@entry=0x2ab43d5325c0) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:241
#15 0x00000000007a5436 in execute (this=0x2ab43d5325c0) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#16 EThread::execute (this=0x2ab43d5325c0) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#17 0x00000000007a3669 in spawn_thread_internal (a=0x2ab4397a3b80) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
#18 0x00002ab437bb4ea5 in start_thread () from /lib64/libpthread.so.0
#19 0x00002ab4388ea8dd in clone () from /lib64/libc.so.6
```
```
(gdb) bt
#0 Mutex_lock (t=0x2ba9791b04c0, m=0x0) at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Lock.h:446
#1 Mutex_lock (t=0x2ba9791b04c0, m=...) at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Lock.h:358
#2 MutexLock::MutexLock (this=0x2ba97d0750f0, am=..., t=0x2ba9791b04c0)
at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Lock.h:446
#3 0x000000000060f70b in Http2Stream::initiating_close() ()
at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/include/tscore/ink_thread.h:124
#4 0x00000000006091f2 in Http2ConnectionState::cleanup_streams() () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ConnectionState.cc:1342
#5 0x000000000060bc93 in Http2ConnectionState::main_event_handler(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ConnectionState.cc:1017
#6 0x0000000000604620 in handleEvent (data=0x0, event=2252, this=0x2bac32592900)
at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#7 handleEvent (data=0x0, event=2252, this=0x2bac32592900)
at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#8 rcv_goaway_frame(Http2ConnectionState&, Http2Frame const&) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ConnectionState.cc:756
#9 0x000000000060c1a4 in Http2ConnectionState::main_event_handler(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ConnectionState.cc:1063
#10 0x00000000005ffac9 in handleEvent (data=0x2ba97d075370, event=2253, this=0x2bac32592900)
at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#11 handleEvent (data=0x2ba97d075370, event=2253, this=0x2bac32592900)
at /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#12 send_connection_event(Continuation*, int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:65
#13 0x00000000006004f7 in Http2ClientSession::do_complete_frame_read (this=this@entry=0x2bac32592600) at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:568
#14 0x0000000000600985 in Http2ClientSession::state_process_frame_read(int, VIO*, bool) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:626
#15 0x00000000005ffdeb in Http2ClientSession::main_event_handler(int, void*) () at ../../../../../../_vcs/trafficserver9/proxy/http2/Http2ClientSession.cc:352
#16 0x000000000076327e in read_signal_and_update(int, UnixNetVConnection*) () at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:83
#17 0x0000000000739a69 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) () at ../../../../../../_vcs/trafficserver9/iocore/net/SSLNetVConnection.cc:670
#18 0x0000000000757abe in NetHandler::process_ready_list (this=this@entry=0x2ba9791b4560) at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:413
#19 0x0000000000757f00 in NetHandler::waitForActivity(long) () at ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:548
#20 0x00000000007a51e1 in EThread::execute_regular (this=this@entry=0x2ba9791b04c0) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/I_PriorityEventQueue.h:115
#21 0x00000000007a5436 in execute (this=0x2ba9791b04c0) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#22 EThread::execute (this=0x2ba9791b04c0) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#23 0x00000000007a3669 in spawn_thread_internal (a=0x2ba9765a3880) at ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
#24 0x00002ba974894ea5 in start_thread () from /lib64/libpthread.so.0
#25 0x00002ba9755ca8dd in clone () from /lib64/libc.so.6
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv commented on issue #7333: Crashes with 9.0.x
Posted by GitBox <gi...@apache.org>.
sudheerv commented on issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333#issuecomment-730666333
Hmm, wait T20 should be locking it with the read vio's mutex which means T11 can't get the lock until it finishes do_io_close?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #7333: Crashes with 9.0.x
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333#issuecomment-730721301
We also got a similar asan that was reported as a double free on the netvc object.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich edited a comment on issue #7333: Crashes with 9.0.x
Posted by GitBox <gi...@apache.org>.
shinrich edited a comment on issue #7333:
URL: https://github.com/apache/trafficserver/issues/7333#issuecomment-730663513
Got an ASAN use-after free when running in our prod-sim environment.
It looks like T20 sets the close flag on the migrated netvc object at the start of do_io_close, but is still working with it. In the meantime, T11 sees the close flag and deletes the migrated netvc from underneath it.
I can see how that causes a use after free. I don't immediately see how that would cause the memory corruptions we are seeing. I have another build running in production.
```
NOTE: using command line path as RUNROOT
Traffic Server 9.0.0.89 Nov 19 2020 21:01:20 atslab02.champaign.office.oath
traffic_server: using root directory '/home/y'
[2020-11-19 21:16:46][INFO ][_vcs/iplookup_cpp/src/IpLookup.cpp][init:88] Loading /home/y/libdata/geotargeting/datapacks/coregeo_ipdatapack_country
[2020-11-19 21:16:46][INFO ][_vcs/iplookup_cpp/src/IpLookup.cpp][init:124] Memory mapping - No preload./home/y/libdata/geotargeting/datapacks/coregeo_ipdatapack_country
[2020-11-19 21:16:46][INFO ][_vcs/iplookup_cpp/src/IpLookup.cpp][init:88] Loading /home/y/libdata/geotargeting/datapacks/coregeo_ipdatapack_country_ipv6
[2020-11-19 21:16:46][INFO ][_vcs/iplookup_cpp/src/IpLookup.cpp][init:124] Memory mapping - No preload./home/y/libdata/geotargeting/datapacks/coregeo_ipdatapack_country_ipv6
=================================================================
==10253==ERROR: AddressSanitizer: heap-use-after-free on address 0x61a00008bad0 at pc 0x000000ed60a8 bp 0x2ba09f4e2a40 sp 0x2ba09f4e2a30
WRITE of size 8 at 0x61a00008bad0 thread T20 ([ET_NET 18])
#0 0xed60a7 in UnixNetVConnection::do_io_close(int) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:677
#1 0xe3cee5 in SSLNetVConnection::do_io_close(int) ../../../../../../_vcs/trafficserver9/iocore/net/SSLNetVConnection.cc:918
#2 0xef03bc in UnixNetVConnection::migrateToCurrentThread(Continuation*, EThread*) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:1470
#3 0x873f98 in HttpSessionManager::acquire_session(sockaddr const*, ats::CryptoHash const&, HttpSM*, TSServerSessionSharingMatchMask, TSServerSessionSharingPoolType) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSessionManager.cc:418
#4 0x875457 in HttpSessionManager::acquire_session(Continuation*, sockaddr const*, char const*, ProxyTransaction*, HttpSM*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSessionManager.cc:381
#5 0x843e6d in HttpSM::do_http_server_open(bool) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:4973
#6 0x84f257 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7523
#7 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#8 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#9 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#10 0x2ba0a1669a5c in cb _vcs/quick_filter-9/quick_filter/quick_filter.cc:637
#11 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#12 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#13 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#14 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#15 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#16 0x84f95a in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7360
#17 0x8025a9 in HttpSM::do_hostdb_lookup() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:4180
#18 0x84f518 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7676
#19 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#20 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#21 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#22 0x2ba0a947bf6c in main_handler ../../../../../_vcs/trafficserver9/plugins/regex_revalidate/regex_revalidate.c:453
#23 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#24 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#25 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#26 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#27 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#28 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#29 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#30 0x2ba0a87d41cc in carpLookup(tsapi_cont*, TSEvent, void*) _vcs/carp-9/carp/carp.cc:767
#31 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#32 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#33 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#34 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#35 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#36 0x82d21f in HttpSM::setup_cache_lookup_complete_api() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2516
#37 0x82d21f in HttpSM::state_cache_open_read(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2578
#38 0x82f3dc in HttpSM::main_handler(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2620
#39 0x9506fc in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#40 0x9506fc in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#41 0x9506fc in HttpCacheSM::state_cache_open_read(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:133
#42 0xca38e1 in Cache::open_read(Continuation*, ats::CryptoHash const*, HTTPHdr*, OverridableHttpConfigParams const*, CacheFragType, char const*, int) ../../../../../../_vcs/trafficserver9/iocore/cache/CacheRead.cc:149
#43 0xc203ab in CacheProcessor::open_read(Continuation*, HttpCacheKey const*, HTTPHdr*, OverridableHttpConfigParams const*, long, CacheFragType) ../../../../../../_vcs/trafficserver9/iocore/cache/Cache.cc:3242
#44 0x94fe0b in HttpCacheSM::do_cache_open_read(HttpCacheKey const&) ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:273
#45 0x951186 in HttpCacheSM::open_read(HttpCacheKey const*, URL*, HTTPHdr*, OverridableHttpConfigParams const*, long) ../../../../../../_vcs/trafficserver9/proxy/http/HttpCacheSM.cc:307
#46 0x7f0f10 in HttpSM::do_cache_lookup_and_read() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:4611
#47 0x84ebd8 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7491
#48 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#49 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#50 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#51 0x2ba0a1669a5c in cb _vcs/quick_filter-9/quick_filter/quick_filter.cc:637
#52 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#53 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#54 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#55 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#56 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#57 0x84f95a in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7360
#58 0x84e631 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7742
#59 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#60 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#61 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#62 0x2ba0a6e39bb4 in http_hook _vcs/vz_connection-9/INKPluginInit.cc:427
#63 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#64 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#65 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#66 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#67 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#68 0x84f95a in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7360
#69 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#70 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#71 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#72 0x2ba0a8f54ae1 in health_check_origin ../../../../../_vcs/trafficserver9/plugins/healthchecks/healthchecks.c:541
#73 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#74 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#75 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#76 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#77 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#78 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#79 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#80 0x2ba0a8cf7398 in cont_rewrite_headers ../../../../../_vcs/trafficserver9/plugins/header_rewrite/header_rewrite.cc:308
#81 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#82 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#83 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#84 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#85 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#86 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#87 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#88 0x2ba0a87d633c in onReadRequest(tsapi_cont*, TSEvent, void*) _vcs/carp-9/carp/carp.cc:375
#89 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#90 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#91 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#92 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#93 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#94 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#95 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#96 0x2ba0a6e39bb4 in http_hook _vcs/vz_connection-9/INKPluginInit.cc:427
#97 0x6bc1a6 in INKContInternal::handle_event(int, void*) ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1096
#98 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:167
#99 0x6f7b02 in Continuation::handleEvent(int, void*) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_Continuation.h:163
#100 0x6f7b02 in APIHook::invoke(int, void*) const ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:1333
#101 0x826141 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1480
#102 0x84f95a in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7360
#103 0x816c91 in HttpSM::state_read_client_request_header(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:837
#104 0x82f5be in HttpSM::main_handler(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:2617
#105 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#106 0x830b0f in HttpSM::do_api_callout() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:353
#107 0x830b0f in HttpSM::state_add_to_list(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:382
#108 0x836f2d in HttpSM::attach_client_session(ProxyTransaction*, IOBufferReader*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:558
#109 0x79c007 in Http1ClientSession::state_keep_alive(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/Http1ClientSession.cc:369
#110 0xed76df in read_signal_and_update ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:83
#111 0xe45d14 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) ../../../../../../_vcs/trafficserver9/iocore/net/SSLNetVConnection.cc:670
#112 0xeb0194 in NetHandler::process_ready_list() ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:413
#113 0xeb0f42 in NetHandler::waitForActivity(long) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:548
#114 0xfc80fe in EThread::execute_regular() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:271
#115 0xfc8955 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#116 0xfc8955 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#117 0xfc24ea in spawn_thread_internal ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
#118 0x2ba0948fcdd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
#119 0x2ba09583a02c in __clone (/lib64/libc.so.6+0xfe02c)
0x61a00008bad0 is located 592 bytes inside of 1264-byte region [0x61a00008b880,0x61a00008bd70)
freed by thread T11 ([ET_NET 9]) here:
#0 0x63cd20 in __interceptor_free (/opt/oath/trafficserver/9.0/bin/traffic_server+0x63cd20)
#1 0x2ba092f2007c in ink_freelist_free ../../../../../../_vcs/trafficserver9/src/tscore/ink_queue.cc:281
#2 0xeaf762 in NetHandler::free_netevent(NetEvent*) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:365
#3 0xeb01ba in NetHandler::process_ready_list() ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:428
#4 0xeb0f42 in NetHandler::waitForActivity(long) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNet.cc:548
#5 0xfc80fe in EThread::execute_regular() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:271
#6 0xfc8955 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:332
#7 0xfc8955 in EThread::execute() ../../../../../../_vcs/trafficserver9/iocore/eventsystem/UnixEThread.cc:310
#8 0xfc24ea in spawn_thread_internal ../../../../../../_vcs/trafficserver9/iocore/eventsystem/Thread.cc:92
#9 0x2ba0948fcdd4 in start_thread (/lib64/libpthread.so.0+0x7dd4)
previously allocated by thread T11 ([ET_NET 9]) here:
#0 0x63df00 in __interceptor_posix_memalign (/opt/oath/trafficserver/9.0/bin/traffic_server+0x63df00)
#1 0x2ba092f1e10f in ats_memalign ../../../../../../_vcs/trafficserver9/src/tscore/ink_memory.cc:102
#2 0x2ba092f1ffde in ink_freelist_new ../../../../../../_vcs/trafficserver9/src/tscore/ink_queue.cc:187
#3 0xe2d900 in ClassAllocator<SSLNetVConnection>::alloc() /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/include/tscore/Allocator.h:131
#4 0xe2d900 in SSLNetVConnection* thread_alloc_init<SSLNetVConnection>(ClassAllocator<SSLNetVConnection>&, ProxyAllocator&) /home/shinrich/build/_build/build_release_posix-x86_64_gcc/trafficserver9/build/../../../../_vcs/trafficserver9/iocore/eventsystem/I_ProxyAllocator.h:73
#5 0xe2d900 in SSLNetProcessor::allocate_vc(EThread*) ../../../../../../_vcs/trafficserver9/iocore/net/SSLNetProcessor.cc:107
#6 0xef02ec in UnixNetVConnection::migrateToCurrentThread(Continuation*, EThread*) ../../../../../../_vcs/trafficserver9/iocore/net/UnixNetVConnection.cc:1448
#7 0x873f98 in HttpSessionManager::acquire_session(sockaddr const*, ats::CryptoHash const&, HttpSM*, TSServerSessionSharingMatchMask, TSServerSessionSharingPoolType) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSessionManager.cc:418
#8 0x875457 in HttpSessionManager::acquire_session(Continuation*, sockaddr const*, char const*, ProxyTransaction*, HttpSM*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSessionManager.cc:381
#9 0x843e6d in HttpSM::do_http_server_open(bool) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:4973
#10 0x84f257 in HttpSM::set_next_state() ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:7523
#11 0x826531 in HttpSM::state_api_callout(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1552
#12 0x831725 in HttpSM::state_api_callback(int, void*) ../../../../../../_vcs/trafficserver9/proxy/http/HttpSM.cc:1351
#13 0x6fef64 in TSHttpTxnReenable ../../../../../_vcs/trafficserver9/src/traffic_server/InkAPI.cc:6164
#14 0x2ba0a1669a5c in cb _vcs/quick_filter-9/quick_filter/quick_filter.cc:637
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org