[jira] [Commented] (TS-3522) Seg Fault due to inactivity_cop after lost continutation from write_signal_and_update

["Alan M. Carroll (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TS-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500065#comment-14500065 ] 

Alan M. Carroll commented on TS-3522:
-------------------------------------

Looking at a core dump from this on 5.3.0.

The problem is a {{UnixNetVConnection}} where {{write.vio._cont}} is dead memory ({{0xdeadbeef}}). The transaction stalls and then InactivityCop tries to shut down the netVC. This falls through and triggers the write signalling which explodes on contact with the bad continuation. This is just a symptom, however, the real problem is the bad continuation being in the VIO.

In a case I dug in to the state machine is still alive. The transaction was a {{POST}} (http -> http) and the state machine had apparently sent the request and post data to the origin and was waiting for a response. I couldn't find how the write VIO continuation could become dead memory because all of the {{do_io_write}} calls seem to use the {{HttpSM}} as the continuation which is clearly not dead memory (the write VIO continuation pointer doesn't seem to match any of the relevant objects).

Other anomalies are that {{from_accept_thread}} is {{true}} which should not be the case. The IP address in the {{con}} member doesn't match the request. The {{action_}} continuation handler is {{SSLNextProtocolAccept::mainEvent}}. I need to check if this is appropriate for a non-SSL connection (this may be a standard way to do the protocol detect). On the other hand this looks like there is some possibly bad sharing going on where this connection is being used for an origin server and user agent connection.

> Seg Fault due to inactivity_cop after lost continutation from write_signal_and_update
> -------------------------------------------------------------------------------------
>
>                 Key: TS-3522
>                 URL: https://issues.apache.org/jira/browse/TS-3522
>             Project: Traffic Server
>          Issue Type: Bug
>          Components: Network
>            Reporter: Steven Feltner
>            Assignee: Alan M. Carroll
>             Fix For: 6.0.0
>
>
> (gdb) bt full
> #0  0x00000000006ec51e in handleEvent (event=105, vc=0x2b1c900461e0) at ../../iocore/eventsystem/I_Continuation.h:146
> No locals.
> #1  write_signal_and_update (event=105, vc=0x2b1c900461e0) at UnixNetVConnection.cc:154
> No locals.
> #2  0x00000000006ec837 in UnixNetVConnection::mainEvent (this=0x2b1c900461e0, event=<value optimized out>, e=<value optimized out>) at UnixNetVConnection.cc:1089
>         wlock = {m = {m_ptr = 0x2b1c90117dd0}, lock_acquired = true}
>         signal_event = 105
>         next_activity_timeout_at = 0
>         t = 0x0
>         hlock = {m = {m_ptr = 0x1430c30}, lock_acquired = true}
>         rlock = {m = {m_ptr = 0x2b1c90117dd0}, lock_acquired = true}
>         signal_timeout = 0x2b1c6b9ddc30
>         reader_cont = 0x0
>         writer_cont = 0x2b1d28051d48
>         signal_timeout_at = 0x2b1c900463f8
> #3  0x00000000006e5061 in handleEvent (this=0x14519d0, event=<value optimized out>, e=0x15792d0) at ../../iocore/eventsystem/I_Continuation.h:146
> No locals.
> #4  InactivityCop::check_inactivity (this=0x14519d0, event=<value optimized out>, e=0x15792d0) at UnixNet.cc:80
>         vc = 0x2b1c900461e0
>         lock = {m = {m_ptr = 0x2b1c90117dd0}, lock_acquired = true}
>         now = 1428965697221995775
>         nh = 0x2b1c695bea30
>         __func__ = "check_inactivity"
> #5  0x000000000070f628 in handleEvent (this=0x2b1c695bb010, e=0x15792d0, calling_code=2) at I_Continuation.h:146
> No locals.
> #6  EThread::process_event (this=0x2b1c695bb010, e=0x15792d0, calling_code=2) at UnixEThread.cc:144
>         c_temp = 0x14519d0
>         lock = {m = {m_ptr = 0x1430c30}, lock_acquired = true}
> #7  0x00000000007101c1 in EThread::execute (this=0x2b1c695bb010) at UnixEThread.cc:223
>         done_one = true
>         e = <value optimized out>
>         NegativeQueue = {<DLL<Event, Event::Link_link>> = {head = 0x1579330}, tail = 0x1579330}
>         next_time = 1428963217761407178
> #8  0x000000000070ea52 in spawn_thread_internal (a=0x144a330) at Thread.cc:88
>         p = 0x144a330
> #9  0x000000383e8079d1 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #10 0x000000383e0e88fd in clone () from /lib64/libc.so.6
> No symbol table info available.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)