You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by GitBox <gi...@apache.org> on 2020/06/20 20:14:38 UTC
[GitHub] [trafficserver] sudheerv opened a new issue #6932: Crash possibly related to PR 5952 and 6686.
sudheerv opened a new issue #6932:
URL: https://github.com/apache/trafficserver/issues/6932
After a week of working through and resolving more frequent crashes in prod, we ended up with hopefully one of the last remaining slow burn crashes. This crash would happen once or twice a day out-of-nowhere after seemingly running fine until then indicating that this is related to some sort of unique request or some other asynchronous/background activity. Digging deeper, I've isolated that to a spike in the metric `proxy.process.hostdb.cache.total_inserts` - Each of those odd crashes (we've ats9 running in a box in each of our DCs) always seem to follow a noticeable spike in hostdb inserts a minute before the crash.
That led me to look through all the recent HostDB changes which may be causing the issue and to isolate further, I've reverted PR #5952 (and the follow-up PR #6686 ) which seems to have fixed the issue. Will need to run this through the weekend to confirm but with those two PRs reverted, haven't seen any crash since yesterday.
Below are some example stack traces that were seen. In most cases, the stack traces don't seem to give much to go on, as they are only symptoms of what appears to be some sort of memory corruption.
```
(gdb) bt
#0 0x00002b1c1476106f in _Unwind_IteratePhdrCallback (info=<optimized out>, size=<optimized out>, ptr=0x2b1c21901980) at ../.././libgcc/unwind-dw2-fde-dip.c:398
#1 0x00002b1c14aa242c in dl_iterate_phdr () from /lib64/libc.so.6
#2 0x00002b1c14761501 in _Unwind_Find_FDE (pc=0x2b1c16d38e81 <lixer::LixPlugin::handleReadRequestHeadersPreRemap(atscppapi::Transaction&)+845>, bases=bases@entry=0x2b1c21901b08) at ../.././libgcc/unwind-dw2-fde-dip.c:469
#3 0x00002b1c1475da43 in uw_frame_state_for (context=context@entry=0x2b1c21901a60, fs=fs@entry=0x2b1c21901b50) at ../.././libgcc/unwind-dw2.c:1249
#4 0x00002b1c1475f988 in _Unwind_Backtrace (trace=0x2b1c14a79da0 <backtrace_helper>, trace_argument=0x2b1c21901d10) at ../.././libgcc/unwind.inc:290
#5 0x00002b1c14a79f16 in backtrace () from /lib64/libc.so.6
#6 0x00002b1c12064a43 in ink_stack_trace_dump () at ink_stack_trace.cc:63
#7 0x00002b1c120798b3 in signal_crash_handler (signo=signo@entry=11) at signals.cc:180
#8 0x00000000004c42de in crash_logger_invoke (signo=11, info=0x2b1c219022b0, ctx=0x2b1c21902180) at traffic_server/Crash.cc:173
#9 <signal handler called>
#10 0x00000000004cf935 in ~Ptr (this=0x2b1c21902e70, __in_chrg=<optimized out>) at ../include/tscore/Ptr.h:215
#11 MutexTryLock::~MutexTryLock (this=0x2b1c21902e70, __in_chrg=<optimized out>) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Lock.h:550
#12 0x00000000004ecd02 in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6097
#13 0x00002b1c2f4df5ad in UreGenerator::handleReadRequestHeadersPreRemap (this=0x2b1c15fffa80, transaction=...) at /export/content/data/multiproduct-post-commit/i001/workspace/ats-plugin-userrequestevent_78f261cbcb2e5016edb9902f1accbb4e89878e6f/ats-plugin-userrequestevent/main/src/UreGenerator.cc:311
#14 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
#15 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f53ba0, event=60016, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#16 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60016, this=0x2b1c17f53ba0) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#17 APIHook::invoke (this=this@entry=0x2b1c15d75ec0, event=60016, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#18 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#19 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#20 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#21 0x00002b1c16d38e82 in lixer::LixPlugin::handleReadRequestHeadersPreRemap (this=0x2b1c17bcafc0, transaction=...) at lixer.cc:127
#22 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
#23 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f53e20, event=60016, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#24 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60016, this=0x2b1c17f53e20) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#25 APIHook::invoke (this=this@entry=0x2b1c15d75ee0, event=60016, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#26 0x000000000054ef77 in HttpSM::state_api_callout (this=0x2b1ccb9c0490, event=event@entry=0, data=data@entry=0x0) at HttpSM.cc:1500
#27 0x000000000054f8df in HttpSM::do_api_callout_internal (this=<optimized out>) at HttpSM.cc:5335
#28 0x000000000055e4b5 in HttpSM::do_api_callout (this=this@entry=0x2b1ccb9c0490) at HttpSM.cc:365
#29 0x00000000005588b3 in HttpSM::set_next_state (this=0x2b1ccb9c0490) at HttpSM.cc:7408
#30 0x00000000005433f2 in HttpSM::call_transact_and_set_next_state (this=this@entry=0x2b1ccb9c0490, f=f@entry=0x0) at HttpSM.cc:7374
#31 0x000000000055392a in HttpSM::handle_api_return (this=0x2b1ccb9c0490) at HttpSM.cc:1635
#32 0x000000000054f4ae in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1567
#33 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#34 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#35 0x00002b1c417f72b6 in setSceLixValue (contp=0x2b1c56111560, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at HttpDebugger.cc:109
#36 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c56111560, event=60002, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#37 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60002, this=0x2b1c56111560) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#38 APIHook::invoke (this=this@entry=0x2b1c15d75ac0, event=60002, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#39 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#40 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#41 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#42 0x00002b1c3ac2d3b2 in cont_rewrite_headers (contp=0x2b1c17f52f20, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at header_rewrite.cc:332
#43 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f52f20, event=60002, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#44 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60002, this=0x2b1c17f52f20) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#45 APIHook::invoke (this=this@entry=0x2b1c15d75ce0, event=60002, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#46 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#47 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#48 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#49 0x00002b1c3a977ff1 in dc_selection::EntityRoutingPlugin::handleReadRequestHeaders (this=0x2b1c15c210c0, transaction=...) at EntityRoutingPlugin.cc:342
#50 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
#51 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f53060, event=60002, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
#52 0x00000000004ea60b in handleEvent (data=0x2b1ccb9c0490, event=60002, this=0x2b1c17f53060) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#53 APIHook::invoke (this=this@entry=0x2b1c15d75d00, event=60002, edata=edata@entry=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1333
#54 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#55 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b1ccb9c0490, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#56 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b1ccb9c0490, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#57 0x00002b1c340a9bf9 in TraceHeadersPlugin::handleReadRequestHeaders (this=0x2b1c15c06bc0, transaction=...) at trace-headers.cc:875
#58 0x00002b1c2bc14c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b1ccb9c0490) at GlobalPlugin.cc:65
#59 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b1c17f53a60, event=60002, edata=0x2b1ccb9c0490) at traffic_server/InkAPI.cc:1096
---Type <return> to continue, or q <return> to quit---q
```
```
#0 ssl_callback_session_ticket (ssl=0x2b588f91e000, keyname=0x2b57d2f043f0 "\260D\360\322W+", iv=0x2b57d2f043e0 "", cipher_ctx=0x2b585887cd40, hctx=0x2b582398e700, enc=1) at SSLSessionTicket.cc:67
67 SSLSessionTicket.cc: No such file or directory.
(gdb) bt
#0 ssl_callback_session_ticket (ssl=0x2b588f91e000, keyname=0x2b57d2f043f0 "\260D\360\322W+", iv=0x2b57d2f043e0 "", cipher_ctx=0x2b585887cd40, hctx=0x2b582398e700, enc=1) at SSLSessionTicket.cc:67
#1 0x00002b57c96a1718 in tls_construct_new_session_ticket () from /lib/libssl.so.1.1
#2 0x00002b57c96933f7 in state_machine () from /lib/libssl.so.1.1
#3 0x00002b57c9668dea in ssl3_read_bytes () from /lib/libssl.so.1.1
#4 0x00002b57c96708fa in ssl3_read () from /lib/libssl.so.1.1
#5 0x00002b57c967c07d in ssl_read_internal () from /lib/libssl.so.1.1
#6 0x00002b57c967c253 in SSL_read () from /lib/libssl.so.1.1
#7 0x0000000000754315 in SSLReadBuffer (ssl=0x2b588f91e000, buf=0x2b589ce95000, nbytes=nbytes@entry=4096, nread=@0x2b57d2f04968: 0) at SSLUtils.cc:1851
#8 0x00000000007439fa in ssl_read_from_net (ret=<synthetic pointer>, lthread=<optimized out>, sslvc=0x2b58dcf86330) at SSLNetVConnection.cc:275
#9 SSLNetVConnection::net_read_io (this=0x2b58dcf86330, nh=0x2b57ce223d80, lthread=<optimized out>) at SSLNetVConnection.cc:665
#10 0x0000000000762498 in NetHandler::process_ready_list (this=this@entry=0x2b57ce223d80) at UnixNet.cc:412
#11 0x000000000076278d in NetHandler::waitForActivity (this=0x2b57ce223d80, timeout=<optimized out>) at UnixNet.cc:547
#12 0x00000000007c6a9a in EThread::execute_regular (this=this@entry=0x2b57ce220000) at UnixEThread.cc:266
#13 0x00000000007c6d62 in EThread::execute (this=0x2b57ce220000) at UnixEThread.cc:327
#14 0x00000000007c5109 in spawn_thread_internal (a=0x2b57cc56de80) at Thread.cc:92
#15 0x00002b57ca315dd5 in start_thread () from /lib64/libpthread.so.0
#16 0x00002b57cb0c6ead in clone () from /lib64/libc.so.6
(gdb) info thr
Id Target Id Frame
```
```
#9 <signal handler called>
#10 std::_Rb_tree_insert_and_rebalance (__insert_left=<optimized out>, __x=0x2b4c234e0fa0, __p=<optimized out>, __header=...) at ../../../.././libstdc++-v3/src/c++98/tree.cc:237
#11 0x00002b4c2c3528aa in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::_M_insert_<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::_Alloc_node>(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&&, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::_Alloc_node&) (this=0x2b4c2350bb20, __x=0x0, __p=0x2b4c224e0d00, __v=<unknown type in /lib64/libtrafficserver-plugin-utils.so, CU 0x3ec0e, DIE 0x730ae>, __node_gen=...) at /usr/include/c++/7.3.0/bits/stl_tree.h:1755
#12 0x00002b4c2c350df2 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::_M_insert_unique<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&&) (this=0x2b4c2350bb20, __v=<unknown type in /lib64/libtrafficserver-plugin-utils.so, CU 0x3ec0e, DIE 0x756e9>) at /usr/include/c++/7.3.0/bits/stl_tree.h:2096
#13 0x00002b4c2c34f124 in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, AtsPluginUtils::CaseInsensitiveStringComparator, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >::insert(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::list<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&&) (this=0x2b4c2350bb20, __x=<unknown type in /lib64/libtrafficserver-plugin-utils.so, CU 0x3ec0e, DIE 0x77e21>) at /usr/include/c++/7.3.0/bits/stl_map.h:803
#14 0x00002b4c2c34b442 in AtsPluginUtils::ClientRequestImpl::appendHeaderField (this=0x2b4c2350ba00, name=..., value=...) at ClientRequestImpl.cc:565
#15 0x00002b4c2c34b310 in AtsPluginUtils::ClientRequestImpl::setHeaderField (this=0x2b4c2350ba00, name=..., value=...) at ClientRequestImpl.cc:555
#16 0x00002b4c2c3474f9 in AtsPluginUtils::ClientRequest::setHeaderField (this=0x2b4c235fe8e0, name=..., value=...) at ClientRequest.cc:171
#17 0x00002b4c2f84c5a3 in PluginControlInternals::addPluginToSignalHeader (pluginName=..., clientRequest=..., pLogger=0x2b4c2422c100) at PluginHeaderSignal.cc:137
#18 0x00002b4c2f83f7f6 in PluginControls::canIrun (this=0x2b4c20503350) at PluginControls.cc:564
#19 0x00002b4c2f30f2ee in UreGenerator::handleReadRequestHeadersPreRemap (this=0x2b4c15fffb80, transaction=...) at /export/content/data/multiproduct-post-commit/i001/workspace/ats-plugin-userrequestevent_78f261cbcb2e5016edb9902f1accbb4e89878e6f/ats-plugin-userrequestevent/main/src/UreGenerator.cc:280
#20 0x00002b4c2ba44c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b4c25107950) at GlobalPlugin.cc:65
#21 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b4c17b7bba0, event=60016, edata=0x2b4c25107950) at traffic_server/InkAPI.cc:1096
#22 0x00000000004ea60b in handleEvent (data=0x2b4c25107950, event=60016, this=0x2b4c17b7bba0) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#23 APIHook::invoke (this=this@entry=0x2b4c15d75ec0, event=60016, edata=edata@entry=0x2b4c25107950) at traffic_server/InkAPI.cc:1333
#24 0x000000000054ef77 in HttpSM::state_api_callout (this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1500
#25 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#26 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b4c25107950, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#27 0x00002b4c2b81de82 in lixer::LixPlugin::handleReadRequestHeadersPreRemap (this=0x2b4c16ff2fc0, transaction=...) at lixer.cc:127
#28 0x00002b4c2ba44c07 in (anonymous namespace)::handleGlobalPluginEvents (cont=<optimized out>, event=TS_EVENT_HTTP_PRE_REMAP, edata=0x2b4c25107950) at GlobalPlugin.cc:65
#29 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b4c17b7be20, event=60016, edata=0x2b4c25107950) at traffic_server/InkAPI.cc:1096
#30 0x00000000004ea60b in handleEvent (data=0x2b4c25107950, event=60016, this=0x2b4c17b7be20) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#31 APIHook::invoke (this=this@entry=0x2b4c15d75ee0, event=60016, edata=edata@entry=0x2b4c25107950) at traffic_server/InkAPI.cc:1333
#32 0x000000000054ef77 in HttpSM::state_api_callout (this=0x2b4c25107950, event=event@entry=0, data=data@entry=0x0) at HttpSM.cc:1500
#33 0x000000000054f8df in HttpSM::do_api_callout_internal (this=<optimized out>) at HttpSM.cc:5335
#34 0x000000000055e4b5 in HttpSM::do_api_callout (this=this@entry=0x2b4c25107950) at HttpSM.cc:365
#35 0x00000000005588b3 in HttpSM::set_next_state (this=0x2b4c25107950) at HttpSM.cc:7408
#36 0x00000000005433f2 in HttpSM::call_transact_and_set_next_state (this=this@entry=0x2b4c25107950, f=f@entry=0x0) at HttpSM.cc:7374
#37 0x000000000055392a in HttpSM::handle_api_return (this=0x2b4c25107950) at HttpSM.cc:1635
38 0x000000000054f4ae in HttpSM::state_api_callout (this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1567
#39 0x0000000000552734 in HttpSM::state_api_callback (this=this@entry=0x2b4c25107950, event=event@entry=60000, data=data@entry=0x0) at HttpSM.cc:1367
#40 0x00000000004eccfa in TSHttpTxnReenable (txnp=0x2b4c25107950, event=TS_EVENT_HTTP_CONTINUE) at traffic_server/InkAPI.cc:6096
#41 0x00002b4c41bf72b6 in setSceLixValue (contp=0x2b4c54d3a560, event=TS_EVENT_HTTP_READ_REQUEST_HDR, edata=0x2b4c25107950) at HttpDebugger.cc:109
#42 0x00000000004d85a1 in INKContInternal::handle_event (this=0x2b4c54d3a560, event=60002, edata=0x2b4c25107950) at traffic_server/InkAPI.cc:1096
#43 0x00000000004ea60b in handleEvent (data=0x2b4c25107950, event=60002, this=0x2b4c54d3a560) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
---Type <return> to continue, or q <return> to quit---q
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #6932: Crash possibly related to PR 5952 and 6686.
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #6932:
URL: https://github.com/apache/trafficserver/issues/6932#issuecomment-647836269
Can you share anything about your DNS/hostdb configuration? Are you using dedicated DNS threads? We spent quite a bit a time tracking down left over events on stale HostDBContinuations. These PRs seemed to address those issues for us.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich edited a comment on issue #6932: Crash possibly related to PR 5952 and 6686.
Posted by GitBox <gi...@apache.org>.
shinrich edited a comment on issue #6932:
URL: https://github.com/apache/trafficserver/issues/6932#issuecomment-647836269
Can you share anything about your DNS/hostdb configuration? Are you using dedicated DNS threads? Are you using SRV records? Or the iterator interface?
We spent quite a bit a time tracking down left over events on stale HostDBContinuations. These PRs seemed to address those issues for us.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv closed issue #6932: Crash possibly related to PR 5952 and 6686.
Posted by GitBox <gi...@apache.org>.
sudheerv closed issue #6932:
URL: https://github.com/apache/trafficserver/issues/6932
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv commented on issue #6932: Crash possibly related to PR 5952 and 6686.
Posted by GitBox <gi...@apache.org>.
sudheerv commented on issue #6932:
URL: https://github.com/apache/trafficserver/issues/6932#issuecomment-649199768
@shinrich Turns out this was a red herring. After 2 weekend days of appearing to be stable with these patches reverted the crashes reappeared (likely due to the increased traffic). The corruption seems to be caused by buffer overflow in log filter.
I'll close this issue and track it using #6946
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org