You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by GitBox <gi...@apache.org> on 2020/06/11 17:13:50 UTC
[GitHub] [trafficserver] sudheerv opened a new issue #6887: crash due to corrupted ssl_hooks
sudheerv opened a new issue #6887:
URL: https://github.com/apache/trafficserver/issues/6887
Seeing crashes due to `ssl_hooks` corrupted. We don't use any plugins that set up ssl_hooks, but, the global variable somehow seems corrupted and crahes.
```
(gdb) thr 58
[Switching to thread 58 (Thread 0x2ae322a05700 (LWP 21621))]
#0 0x00002ae3148177c4 in _dl_addr () from /lib64/libc.so.6
(gdb) bt
#0 0x00002ae3148177c4 in _dl_addr () from /lib64/libc.so.6
#1 0x00002ae3147ef585 in backtrace_symbols_fd () from /lib64/libc.so.6
#2 0x00002ae311dd9a59 in ink_stack_trace_dump () at ink_stack_trace.cc:65
#3 0x00002ae311dee8b3 in signal_crash_handler (signo=signo@entry=11) at signals.cc:180
#4 0x00000000004c434e in crash_logger_invoke (signo=11, info=0x2ae322a03ab0, ctx=0x2ae322a03980) at traffic_server/Crash.cc:173
#5 <signal handler called>
#6 head (this=0x58585858585858b0) at traffic_server/InkAPI.cc:1339
#7 init (id=TS_HTTP_SEND_RESPONSE_HDR_HOOK, feature_hooks=<optimized out>, this=0x2ae36b928578) at traffic_server/InkAPI.cc:1427
#8 HttpHookState::init (this=0x2ae36b928570, id=TS_HTTP_SEND_RESPONSE_HDR_HOOK, global=<optimized out>, ssn=0x2ae3244fe8d8, txn=0x2ae36b9285e0) at traffic_server/InkAPI.cc:1370
#9 0x000000000054e1dd in HttpSM::do_api_callout_internal (this=0x2ae36b9263b0) at HttpSM.cc:5331
#10 0x000000000055c855 in HttpSM::do_api_callout (this=this@entry=0x2ae36b9263b0) at HttpSM.cc:365
#11 0x00000000005505d2 in HttpSM::setup_error_transfer (this=0x2ae36b9263b0) at HttpSM.cc:6433
#12 0x000000000055742d in HttpSM::set_next_state (this=0x2ae36b9263b0) at HttpSM.cc:7697
#13 0x00000000005419b2 in HttpSM::call_transact_and_set_next_state (this=this@entry=0x2ae36b9263b0, f=<optimized out>) at HttpSM.cc:7373
#14 0x000000000054e84e in HttpSM::state_read_server_response_header (this=0x2ae36b9263b0, event=100, data=0x2ae35d4f9fe0) at HttpSM.cc:1962
#15 0x0000000000550d98 in HttpSM::main_handler (this=0x2ae36b9263b0, event=100, data=0x2ae35d4f9fe0) at HttpSM.cc:2710
#16 0x0000000000778215 in handleEvent (data=0x2ae35d4f9fe0, event=100, this=0x2ae36b9263b0) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:193
#17 read_signal_and_update (vc=0x2ae35d4f9e00, event=100) at UnixNetVConnection.cc:83
#18 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2ae35d4f9e00, event=event@entry=100) at UnixNetVConnection.cc:1008
#19 0x000000000074127d in SSLNetVConnection::net_read_io (this=0x2ae35d4f9e00, nh=0x2ae319116d80, lthread=<optimized out>) at SSLNetVConnection.cc:674
#20 0x0000000000760658 in NetHandler::process_ready_list (this=this@entry=0x2ae319116d80) at UnixNet.cc:412
#21 0x000000000076094d in NetHandler::waitForActivity (this=0x2ae319116d80, timeout=<optimized out>) at UnixNet.cc:547
#22 0x00000000007c3eaa in EThread::execute_regular (this=this@entry=0x2ae319113000) at UnixEThread.cc:266
#23 0x00000000007c4112 in EThread::execute (this=0x2ae319113000) at UnixEThread.cc:327
#24 0x00000000007c2499 in spawn_thread_internal (a=0x2ae315d6b300) at Thread.cc:92
#25 0x00002ae313a27dd5 in start_thread () from /lib64/libpthread.so.0
#26 0x00002ae3147d8ead in clone () from /lib64/libc.so.6
(gdb)
(gdb) f 8
#8 HttpHookState::init (this=0x2ae36b928570, id=TS_HTTP_SEND_RESPONSE_HDR_HOOK, global=<optimized out>, ssn=0x2ae3244fe8d8, txn=0x2ae36b9285e0) at traffic_server/InkAPI.cc:1370
1370 traffic_server/InkAPI.cc: No such file or directory.
(gdb) p ssl_hooks
$1 = (SslAPIHooks *) 0x5858585858585858
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv closed issue #6887: crash due to corrupted ssl_hooks
Posted by GitBox <gi...@apache.org>.
sudheerv closed issue #6887:
URL: https://github.com/apache/trafficserver/issues/6887
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv commented on issue #6887: crash due to corrupted ssl_hooks
Posted by GitBox <gi...@apache.org>.
sudheerv commented on issue #6887:
URL: https://github.com/apache/trafficserver/issues/6887#issuecomment-642832248
> @sudheerv is your HttpSM otherwise in good shape?
Yeah, seems so - any particular field you are interested in?
```
#9 0x000000000054e1dd in HttpSM::do_api_callout_internal (this=0x2ae36b9263b0) at HttpSM.cc:5331
5331 HttpSM.cc: No such file or directory.
(gdb) p this->magic
$4 = 65261
(gdb) p this->t_state.request_data
$5 = {<RequestData> = {_vptr.RequestData = 0x81d590 <vtable for HttpRequestData+16>}, hdr = 0x2ae36b926b60, hostname_str = 0x2ae32918cca1 "xxxxxxxxxxxxxxxxxxx", api_info = 0x2ae36b926d20, xact_start = 1591893731, src_ip = {sa = {sa_family = 2,
sa_data = "\026\"\n\362N\225\000\000\000\000\000\000\000"}, sin = {sin_family = 2, sin_port = 8726, sin_addr = {s_addr = 2504978954}, sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 2, sin6_port = 8726, sin6_flowinfo = 2504978954, sin6_addr = {__in6_u = {
__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, dest_ip = {sa = {sa_family = 0, sa_data = '\000' <repeats 13 times>}, sin = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0},
sin_zero = "\000\000\000\000\000\000\000"}, sin6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' <repeats 15 times>, __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}, incoming_port = 12272, tag = 0x0,
internal_txn = false, cache_info_lookup_url = 0x2ae36b9264e0, cache_info_parent_selection_url = 0x2ae36b926540}
(gdb) p this->t_state.m_magic
$6 = HttpTransact::HTTP_TRANSACT_MAGIC_ALIVE
(gdb) p this->history
$7 = {history = {{location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 23}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 23}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e70c0 <HttpSM::state_send_server_request_header(int, void*)::__FUNCTION__> "state_send_server_request_header", line = 2024}, event = 103, reentrancy = 1}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e69f0 <HttpSM::do_setup_post_tunnel(HttpVC_t)::__FUNCTION__> "do_setup_post_tunnel", line = 5891}, event = 34463, reentrancy = 1}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e6d80 <HttpSM::tunnel_handler_post_ua(int, HttpTunnelProducer*)::__FUNCTION__> "tunnel_handler_post_ua",
line = 3558}, event = 102, reentrancy = 0}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e6d30 <HttpSM::tunnel_handler_post_server(int, HttpTunnelConsumer*)::__FUNCTION__> "tunnel_handler_post_server", line = 3659}, event = 103, reentrancy = 0}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e6f10 <HttpSM::tunnel_handler_post(int, void*)::__FUNCTION__> "tunnel_handler_post", line = 2791}, event = 2301, reentrancy = 1}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7100 <HttpSM::state_read_server_response_header(int, void*)::__FUNCTION__> "state_read_server_response_header", line = 1866}, event = 100, reentrancy = 1}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout",
line = 1404}, event = 60000, reentrancy = 4}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 5}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 5}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364},
event = 60000, reentrancy = 6}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 6}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 7}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404},
event = 60000, reentrancy = 7}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 1}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 1}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364},
event = 60000, reentrancy = 2}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 2}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 1}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404},
event = 60000, reentrancy = 1}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 2}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 2}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364},
event = 60000, reentrancy = 3}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 3}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 4}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404},
event = 60000, reentrancy = 4}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 5}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 5}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364},
event = 60000, reentrancy = 6}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 6}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 7}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404},
event = 60000, reentrancy = 7}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 8}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 8}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364},
event = 60000, reentrancy = 9}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 9}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 10}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404},
event = 60000, reentrancy = 10}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 11}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 11}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364},
event = 60000, reentrancy = 12}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 12}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 13}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404},
event = 60000, reentrancy = 13}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 14}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 14}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364},
event = 60000, reentrancy = 15}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 15}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 16}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404},
event = 60000, reentrancy = 16}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 17}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 17}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364},
event = 60000, reentrancy = 18}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 18}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 19}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404},
event = 60000, reentrancy = 19}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 20}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 20}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364},
event = 60000, reentrancy = 21}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 21}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e66b8 <HttpSM::set_next_state()::__FUNCTION__> "set_next_state", line = 7497}, event = 34463, reentrancy = 21}, {location = {file = 0x7e579e "HttpSM.cc", func = 0x7e66b8 <HttpSM::set_next_state()::__FUNCTION__> "set_next_state", line = 7543}, event = 34463, reentrancy = 21}, {location = {
file = 0x7e579e "HttpSM.cc", func = 0x7e71b0 <HttpSM::state_api_callback(int, void*)::__FUNCTION__> "state_api_callback", line = 1364}, event = 60000, reentrancy = 22}, {location = {file = 0x7e579e "HttpSM.cc",
func = 0x7e7190 <HttpSM::state_api_callout(int, void*)::__FUNCTION__> "state_api_callout", line = 1404}, event = 60000, reentrancy = 22}}, history_pos = 73}
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #6887: crash due to corrupted ssl_hooks
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #6887:
URL: https://github.com/apache/trafficserver/issues/6887#issuecomment-642830305
@sudheerv is your HttpSM otherwise in good shape?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on issue #6887: crash due to corrupted ssl_hooks
Posted by GitBox <gi...@apache.org>.
shinrich commented on issue #6887:
URL: https://github.com/apache/trafficserver/issues/6887#issuecomment-642834785
Looks like something is running over the global hook pointers. Can you run an ASAN in your environment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv commented on issue #6887: crash due to corrupted ssl_hooks
Posted by GitBox <gi...@apache.org>.
sudheerv commented on issue #6887:
URL: https://github.com/apache/trafficserver/issues/6887#issuecomment-647039270
Ran ASAN in prod (couldn't run it for longer than an hour before maxing out CPU) and our staging/integration environment (over a day, slow burn as the qps is lower) and found a few heap corruption (in our internal plugins and libraries) and heap overflow issues (#6916 ).
After fixing those, the occurrence of this crash dropped significantly (from once every 3-4 hours to 1-2/day).
Will open a separate issue for the findings on the last remaining crashes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org