[jira] [Commented] (TS-3105) Combination of fixes for TS-3084 and TS-3073 causing asserts and segfaults on 5.1 and beyond

["Susan Hinrichs (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TS-3105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14175314#comment-14175314 ] 

Susan Hinrichs commented on TS-3105:
------------------------------------

Update on this.  For the main system I'm working with, I'm no longer seeing the segfaults.  Only the assert.  I've made changes to set the p->p_handler_state in the case of error events when the handler does not set it otherwise.  This has moved the assert to the following.

==> /y/logs/ets/traffic.out <==
FATAL: HttpSM.cc:3111: failed assert `c->vc == ua_session`
/z/ets/bin/traffic_server - STACK TRACE: 
/z/ets/lib/libtsutil.so.5(+0x25197)[0x2b319a175197]
/z/ets/lib/libtsutil.so.5(+0x23def)[0x2b319a173def]
/z/ets/bin/traffic_server(HttpSM::tunnel_handler_ua(int, HttpTunnelConsumer*)+0xad5)[0x591e85]
/z/ets/bin/traffic_server(HttpTunnel::consumer_handler(int, HttpTunnelConsumer*)+0x9e)[0x5de7de]
/z/ets/bin/traffic_server(HttpTunnel::main_handler(int, void*)+0x117)[0x5ded57]
/z/ets/bin/traffic_server(UnixNetVConnection::mainEvent(int, Event*)+0x3f0)[0x726930]
/z/ets/bin/traffic_server(InactivityCop::check_inactivity(int, Event*)+0x275)[0x7181f5]
/z/ets/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x746591]
/z/ets/bin/traffic_server(EThread::execute()+0x2fb)[0x746e4b]
/z/ets/bin/traffic_server(main+0xcf8)[0x491308]
/lib64/libc.so.6(__libc_start_main+0xf0)[0x2b319c3f8fd0]
/z/ets/bin/traffic_server[0x498131]

Based on warning prints I inserted right before the assert, it looks like the inactivity timer is calling clean up on a VC that has already had it's state machine kill'ed.  If the in_tunnel is still true when kill_this is called on the state machine, the VC's will be left.  In this particular case, the VC was left with some unexpected variables set to null thus causing the assert.

> Combination of fixes for TS-3084 and TS-3073 causing asserts and segfaults on 5.1 and beyond
> --------------------------------------------------------------------------------------------
>
>                 Key: TS-3105
>                 URL: https://issues.apache.org/jira/browse/TS-3105
>             Project: Traffic Server
>          Issue Type: Bug
>            Reporter: Susan Hinrichs
>            Assignee: Susan Hinrichs
>
> These two patches were run in a production environment on top of 5.0.1 without problem for several weeks.  Now running with these patches on top of 5.1 causes either an assert or a segfault.  Another person has reported the same segfault when running master in a production environment.
> In the assert, the handler_state of the producers is 0 (UNKNOWN) rather than a terminal state which is expected.  I'm assuming either we are being directed into the terminal state from a connection that terminates too quickly.  Or an event has hung around for too long and is being executed against the state machine after it has been recycled.
> The event is HTTP_TUNNEL_EVENT_DONE
> The assert stack trace is
> FATAL: HttpSM.cc:2632: failed assert `0`
> /z/bin/traffic_server - STACK TRACE:
> /z/lib/libtsutil.so.5(+0x25197)[0x2b8bd08dc197]
> /z/lib/libtsutil.so.5(+0x23def)[0x2b8bd08dadef]
> /z/bin/traffic_server(HttpSM::tunnel_handler_post_or_put(HttpTunnelProducer*)+0xcd)[0x5982ad]
> /z/bin/traffic_server(HttpSM::tunnel_handler_post(int, void*)+0x86)[0x5a32d6]
> /z/bin/traffic_server(HttpSM::main_handler(int, void*)+0xd8)[0x5a1e18]
> /z/bin/traffic_server(HttpTunnel::main_handler(int, void*)+0xee)[0x5dd6ae]
> /z/bin/traffic_server(write_to_net_io(NetHandler*, UnixNetVConnection*, EThread*)+0x136e)[0x721d1e]
> /z/bin/traffic_server(NetHandler::mainNetEvent(int, Event*)+0x28c)[0x7162fc]
> /z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x744df1]
> /z/bin/traffic_server(EThread::execute()+0x4fc)[0x7458ac]
> /z/bin/traffic_server[0x7440ca]
> /lib64/libpthread.so.0(+0x7034)[0x2b8bd1ee4034]
> /lib64/libc.so.6(clone+0x6d)[0x2b8bd2c2875d]
> The segfault stack trace is 
> /z/bin/traffic_server - STACK TRACE: 
> /lib64/libpthread.so.0(+0xf280)[0x2abccd0d8280]
> /z/bin/traffic_server(HttpSM::tunnel_handler_ua(int, HttpTunnelConsumer*)+0x122)[0x591462]
> /z/bin/traffic_server(HttpTunnel::consumer_handler(int, HttpTunnelConsumer*)+0x9e)[0x5dd15e]
> /z/bin/traffic_server(HttpTunnel::main_handler(int, void*)+0x117)[0x5dd6d7]
> /z/bin/traffic_server(UnixNetVConnection::mainEvent(int, Event*)+0x3f0)[0x725190]
> /z/bin/traffic_server(InactivityCop::check_inactivity(int, Event*)+0x275)[0x716b75]
> /z/bin/traffic_server(EThread::process_event(Event*, int)+0x91)[0x744df1]
> /z/bin/traffic_server(EThread::execute()+0x2fb)[0x7456ab]
> /z/bin/traffic_server[0x7440ca]
> /lib64/libpthread.so.0(+0x7034)[0x2abccd0d0034]
> /lib64/libc.so.6(clone+0x6d)[0x2abccde1475d]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)