You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by GitBox <gi...@apache.org> on 2020/06/05 01:27:01 UTC
[GitHub] [trafficserver] sudheerv opened a new issue #6853: Crash on dynamic_cast
sudheerv opened a new issue #6853:
URL: https://github.com/apache/trafficserver/issues/6853
Seeing this crash in ats9 in prod.
It's a bit strange that `dynamic_cast` would crash this way. `Http2ClientSession` is no longer a subclass of `PluginIdentity` and only `PluginVC` now is derived from `PluginIdentity`, but it's odd that instead of returning a nullptr, dynamic_cast is causing a SegFault on attempting to do that.
Everything else about the pointer seems fine (i.e it seems to be a valid SSL VC pointer). Given that, this dynamic_cast is not required anymore (ie no real netvc is of type `PluginIdentity` anymore, I'm going to try just guarding it with `is_internal_request()` check to see if it helps. It's still pretty strange that it's crashing like this though.
```
(gdb) bt
#0 0x00002b88ea38987e in __cxxabiv1::__dynamic_cast (src_ptr=0x2b89451cb5d0, src_type=0x81dd48 <typeinfo for NetVConnection>, dst_type=dst_type@entry=0x7c49d8 <typeinfo for PluginIdentity>, src2dst=src2dst@entry=-2)
at ../../.././libstdc++-v3/libsupc++/dyncast.cc:71
#1 0x000000000071d0e8 in ProxyTransaction::new_transaction (this=this@entry=0x2b894a65e240, from_early_data=<optimized out>) at ProxyTransaction.cc:46
#2 0x0000000000523c11 in new_transaction (this=0x2b894a65ded0) at Http1ClientSession.cc:473
#3 Http1ClientSession::state_keep_alive (this=0x2b894a65ded0, event=100, data=<optimized out>) at Http1ClientSession.cc:388
#4 0x0000000000770393 in handleEvent (data=0x2b89451cb7b0, event=100, this=0x2b894a65ded0) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:190
#5 read_signal_and_update (vc=0x2b89451cb5d0, event=100) at UnixNetVConnection.cc:83
#6 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2b89451cb5d0, event=event@entry=100) at UnixNetVConnection.cc:1016
#7 0x0000000000741d53 in SSLNetVConnection::net_read_io (this=0x2b89451cb5d0, nh=0x2b88ef623d80, lthread=<optimized out>) at SSLNetVConnection.cc:671
#8 0x0000000000760208 in NetHandler::process_ready_list (this=this@entry=0x2b88ef623d80) at UnixNet.cc:412
#9 0x00000000007604fd in NetHandler::waitForActivity (this=0x2b88ef623d80, timeout=<optimized out>) at UnixNet.cc:547
#10 0x00000000007be9ba in EThread::execute_regular (this=this@entry=0x2b88ef620000) at UnixEThread.cc:266
#11 0x00000000007bec82 in EThread::execute (this=0x2b88ef620000) at UnixEThread.cc:327
#12 0x00000000007bd029 in spawn_thread_internal (a=0x2b88ec163ec0) at Thread.cc:92
#13 0x00002b88e9ee3dd5 in start_thread () from /lib64/libpthread.so.0
#14 0x00002b88eac94ead in clone () from /lib64/libc.so.6
(gdb) p *this
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv commented on issue #6853: Crash on dynamic_cast
Posted by GitBox <gi...@apache.org>.
sudheerv commented on issue #6853:
URL: https://github.com/apache/trafficserver/issues/6853#issuecomment-647038552
Ran ASAN in prod (couldn't run it for longer than an hour before maxing out CPU) and our staging/integration environment (over a day, slow burn as the qps is lower) and found a few heap corruption (in our internal plugins and libraries) and heap overflow issues (#6916 ). After fixing those, the occurrence of this crash dropped significantly (from once every 3-4 hours to 1-2/day).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv closed issue #6853: Crash on dynamic_cast
Posted by GitBox <gi...@apache.org>.
sudheerv closed issue #6853:
URL: https://github.com/apache/trafficserver/issues/6853
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] sudheerv commented on issue #6853: Crash on dynamic_cast
Posted by GitBox <gi...@apache.org>.
sudheerv commented on issue #6853:
URL: https://github.com/apache/trafficserver/issues/6853#issuecomment-640112563
Dug into this some more and it turns out this is likely related to the issue https://github.com/apache/trafficserver/issues/6849 and https://github.com/apache/trafficserver/issues/6850.
The root cause seems to be that the default `max_iobuffer_size` which is supposed to be an iobuffer size index (and not the raw size) and is used in figuring out the iobuffer index for a caller that asks for a IOBuffer (without explicitly passing in a max), is sometimes garbage. This results in corrupting IOBuffers and ends in weird crashes like a few of these.
```
(gdb) p max_iobuffer_size
$1 = 8245894515088831029
```
```
(gdb) bt
#0 0x00002b1926ed987e in __cxxabiv1::__dynamic_cast (src_ptr=0x2b19e1b0baf0, src_type=src_type@entry=0x7c1498 <typeinfo for Continuation>, dst_type=dst_type@entry=0x7c5188 <typeinfo for PluginUserArgsMixin>, src2dst=src2dst@entry=-2) at ../../.././libstdc++-v3/libsupc++/dyncast.cc:71
#1 0x00000000004e17ce in TSUserArgGet (data=<optimized out>, arg_idx=0) at traffic_server/InkAPI.cc:6200
#2 0x00002b193e022144 in atscppapi::utils::internal::getTransaction (ats_txn_handle=ats_txn_handle@entry=0x2b19e1b0baf0) at utils_internal.cc:178
#3 0x00002b193e0230ce in (anonymous namespace)::handleTransactionEvents (cont=0x2b192abf4ec0, event=TS_EVENT_HTTP_READ_RESPONSE_HDR, edata=0x2b19e1b0baf0) at utils_internal.cc:76
#4 0x00000000004d7231 in INKContInternal::handle_event (this=0x2b192abf4ec0, event=60006, edata=0x2b19e1b0baf0) at traffic_server/InkAPI.cc:1096
#5 0x00000000004ef7e6 in Continuation::handleEvent (this=0x2b192abf4ec0, event=event@entry=60006, data=data@entry=0x2b19e1b0baf0) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:193
#6 0x00000000004e9517 in APIHook::invoke (this=this@entry=0x2b1928d6df80, event=60006, edata=edata@entry=0x2b19e1b0baf0) at traffic_server/InkAPI.cc:1333
#7 0x000000000054c987 in HttpSM::state_api_callout (this=0x2b19e1b0baf0, event=<optimized out>, data=<optimized out>) at HttpSM.cc:1499
#8 0x000000000054db4b in HttpSM::state_read_server_response_header (this=0x2b19e1b0baf0, event=100, data=0x2b19f25c9130) at HttpSM.cc:2026
#9 0x000000000054fa58 in HttpSM::main_handler (this=0x2b19e1b0baf0, event=100, data=0x2b19f25c9130) at HttpSM.cc:2729
#10 0x000000000076fbbe in handleEvent (data=0x2b19f25c9130, event=100, this=0x2b19e1b0baf0) at /home/svinukon/Traffic/ATS/ats9/ats-core_trunk/ats9/src/iocore/eventsystem/I_Continuation.h:193
#11 read_signal_and_update (vc=0x2b19f25c8f50, event=100) at UnixNetVConnection.cc:83
#12 UnixNetVConnection::readSignalAndUpdate (this=this@entry=0x2b19f25c8f50, event=event@entry=100) at UnixNetVConnection.cc:1016
#13 0x000000000073f64d in SSLNetVConnection::net_read_io (this=0x2b19f25c8f50, nh=0x2b192aa23d80, lthread=<optimized out>) at SSLNetVConnection.cc:674
#14 0x000000000075de68 in NetHandler::process_ready_list (this=this@entry=0x2b192aa23d80) at UnixNet.cc:412
#15 0x000000000075e15d in NetHandler::waitForActivity (this=0x2b192aa23d80, timeout=<optimized out>) at UnixNet.cc:547
#16 0x00000000007bbc0a in EThread::execute_regular (this=this@entry=0x2b192aa20000) at UnixEThread.cc:266
#17 0x00000000007bbe92 in EThread::execute (this=0x2b192aa20000) at UnixEThread.cc:327
#18 0x00000000007ba1e9 in spawn_thread_internal (a=0x2b1928d63a40) at Thread.cc:92
#19 0x00002b1926a33dd5 in start_thread () from /lib64/libpthread.so.0
#20 0x00002b19277e4ead in clone () from /lib64/libc.so.6
(gdb) p max_iobuffer_size
$1 = 8245894515088831029
(gdb)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org