intermittent crashes in HttpSM::state_http_server_open()

[John Rushford <jr...@apache.org>]

We're running ats 6.2.2 and recently we've seen  a rash of crashes in
HttpSM::state_http_server_open() due to the handler receiving an unexpected
event, VC_EVENT_WRITE_COMPLETE, the handler calls ink_release_assert() in
this case.
The coredumps all show an http POST request to an origin server was in
progress.  These events and coredumps are intermittent and I'm unable to
reproduce the issue in our test environment.  I was wondering if anyone has
seen this before and has an idea why the event occurs.

In the meantime, I have patched this build so that any unexpected events in
this handler send a connection error instead of calling
ink_release_assert().  Here is my patch and would like to know if this
should be pushed upstream to master.

ATS 6.2.x code:

  1 @@ -1759,6 +1759,8 @@ HttpSM::state_http_server_open(int event, void
*data)
  2         do_http_server_open();
  3       }
  4       break;
  5  +  default:
  6  +    Error("[HttpSM::state_http_server_open] Unknown event: %d",
event);
  7     case VC_EVENT_ERROR:
  8     case NET_EVENT_OPEN_FAILED:
  9       t_state.current.state = HttpTransact::CONNECTION_ERROR;
 10  @@ -1793,11 +1795,6 @@ HttpSM::state_http_server_open(int event, void
*data    )
 11       t_state.current.state =
HttpTransact::CONGEST_CONTROL_CONGESTED_ON_M;
 12       call_transact_and_set_next_state(HttpTransact::HandleResponse);
 13       return 0;
 14  -
 15  -  default:
 16  -    Error("[HttpSM::state_http_server_open] Unknown event: %d",
event);
 17  -    ink_release_assert(0);
 18  -    return 0;
 19     }
 20
 21     return 0;

Re: intermittent crashes in HttpSM::state_http_server_open()

[Bryan Call <bc...@apache.org>]

We are not seeing this problem with 7.1.0 in production.  The change does seem reasonable to not assert on an unknown events and would be a good change on master.  I would still leave in an ink_assert() for debug builds.

-Bryan



> On Sep 20, 2017, at 8:40 AM, John Rushford <jr...@apache.org> wrote:
> 
> We're running ats 6.2.2 and recently we've seen  a rash of crashes in
> HttpSM::state_http_server_open() due to the handler receiving an unexpected
> event, VC_EVENT_WRITE_COMPLETE, the handler calls ink_release_assert() in
> this case.
> The coredumps all show an http POST request to an origin server was in
> progress.  These events and coredumps are intermittent and I'm unable to
> reproduce the issue in our test environment.  I was wondering if anyone has
> seen this before and has an idea why the event occurs.
> 
> In the meantime, I have patched this build so that any unexpected events in
> this handler send a connection error instead of calling
> ink_release_assert().  Here is my patch and would like to know if this
> should be pushed upstream to master.
> 
> ATS 6.2.x code:
> 
>  1 @@ -1759,6 +1759,8 @@ HttpSM::state_http_server_open(int event, void
> *data)
>  2         do_http_server_open();
>  3       }
>  4       break;
>  5  +  default:
>  6  +    Error("[HttpSM::state_http_server_open] Unknown event: %d",
> event);
>  7     case VC_EVENT_ERROR:
>  8     case NET_EVENT_OPEN_FAILED:
>  9       t_state.current.state = HttpTransact::CONNECTION_ERROR;
> 10  @@ -1793,11 +1795,6 @@ HttpSM::state_http_server_open(int event, void
> *data    )
> 11       t_state.current.state =
> HttpTransact::CONGEST_CONTROL_CONGESTED_ON_M;
> 12       call_transact_and_set_next_state(HttpTransact::HandleResponse);
> 13       return 0;
> 14  -
> 15  -  default:
> 16  -    Error("[HttpSM::state_http_server_open] Unknown event: %d",
> event);
> 17  -    ink_release_assert(0);
> 18  -    return 0;
> 19     }
> 20
> 21     return 0;