You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@trafficserver.apache.org by GitBox <gi...@apache.org> on 2020/05/21 15:48:27 UTC
[GitHub] [trafficserver] shinrich commented on pull request #6809: Fix assert when client aborts during backfill
shinrich commented on pull request #6809:
URL: https://github.com/apache/trafficserver/pull/6809#issuecomment-632168977
Stack track was
```
* thread #19, name = '[ET_NET 17]', stop reason = signal SIGABRT
frame #0: 0x00007fff7157f33a libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x00007fff7163be60 libsystem_pthread.dylib`pthread_kill + 430
frame #2: 0x00007fff71506808 libsystem_c.dylib`abort + 120
frame #3: 0x000000010ea3a81e libtscore.9.dylib`ink_abort(message_format="%s:%d: failed assertion `%s`") at ink_error.cc:99:3
frame #4: 0x000000010ea3790c libtscore.9.dylib`::_ink_assert(expression="event == HTTP_TUNNEL_EVENT_DONE", file="HttpSM.cc", line=2880) at ink_assert.cc:37:3
* frame #5: 0x000000010d742df6 traffic_server`HttpSM::tunnel_handler(this=0x00007f9959b8f400, event=104, data=0x00007f995558afe8) at HttpSM.cc:2880:3
frame #6: 0x000000010d738e0a traffic_server`HttpSM::main_handler(this=0x00007f9959b8f400, event=104, data=0x00007f995558afe8) at HttpSM.cc:2607:5
frame #7: 0x000000010d65c7cf traffic_server`Continuation::handleEvent(this=0x00007f9959b8f400, event=104, data=0x00007f995558afe8) at I_Continuation.h:190:12
frame #8: 0x000000010dafacc0 traffic_server`read_signal_and_update(event=104, vc=0x00007f995558ae00) at UnixNetVConnection.cc:83:24
frame #9: 0x000000010dafabfd traffic_server`read_signal_done(event=104, nh=0x00007f99588d3d60, vc=0x00007f995558ae00) at UnixNetVConnection.cc:144:7
frame #10: 0x000000010dafabc3 traffic_server`UnixNetVConnection::readSignalDone(this=0x00007f995558ae00, event=104, nh=0x00007f99588d3d60) at UnixNetVConnection.cc:1005:11
frame #11: 0x000000010daa2bad traffic_server`SSLNetVConnection::net_read_io(this=0x00007f995558ae00, nh=0x00007f99588d3d60, lthread=0x00007f99588d0000) at SSLNetVConnection.cc:713:5
frame #12: 0x000000010dae8a83 traffic_server`NetHandler::process_ready_list(this=0x00007f99588d3d60) at UnixNet.cc:412:11
frame #13: 0x000000010dae94d2 traffic_server`NetHandler::waitForActivity(this=0x00007f99588d3d60, timeout=10000000) at UnixNet.cc:547:3
frame #14: 0x000000010db33612 traffic_server`EThread::execute_regular(this=0x00007f99588d0000) at UnixEThread.cc:284:14
frame #15: 0x000000010db33add traffic_server`EThread::execute(this=0x00007f99588d0000) at UnixEThread.cc:345:11
frame #16: 0x000000010db31ba7 traffic_server`spawn_thread_internal(a=0x00007f9953526870) at Thread.cc:92:12
frame #17: 0x00007fff7163c109 libsystem_pthread.dylib`_pthread_start + 148
frame #18: 0x00007fff71637b8b libsystem_pthread.dylib`thread_start + 15
```
History was
```
history = {
[0] = {
location = (file = "HttpSM.cc", func = "state_read_client_request_header", line = 613)
event = 100
reentrancy = 2
}
[1] = {
location = (file = "HttpSM.cc", func = "state_api_callback", line = 1337)
event = 60000
reentrancy = 3
}
[2] = {
location = (file = "HttpSM.cc", func = "state_api_callout", line = 1377)
event = 60000
reentrancy = 3
}
[3] = {
location = (file = "HttpSM.cc", func = "set_next_state", line = 7401)
event = 34463
reentrancy = 3
}
[4] = {
location = (file = "HttpCacheSM.cc", func = "state_cache_open_read", line = 100)
event = 1103
reentrancy = -31073
}
[5] = {
location = (file = "HttpSM.cc", func = "state_cache_open_read", line = 2518)
event = 1103
reentrancy = 4
}
[6] = {
location = (file = "HttpSM.cc", func = "state_api_callback", line = 1337)
event = 60000
reentrancy = 5
}
[7] = {
location = (file = "HttpSM.cc", func = "state_api_callout", line = 1377)
event = 60000
reentrancy = 5
}
[8] = {
location = (file = "HttpSM.cc", func = "state_api_callback", line = 1337)
event = 60000
reentrancy = 6
}
[9] = {
location = (file = "HttpSM.cc", func = "state_api_callout", line = 1377)
event = 60000
reentrancy = 6
}
[10] = {
location = (file = "HttpSM.cc", func = "set_next_state", line = 7361)
event = 34463
reentrancy = 6
}
[11] = {
location = (file = "HttpSM.cc", func = "set_next_state", line = 7510)
event = 34463
reentrancy = 6
}
[12] = {
location = (file = "HttpCacheSM.cc", func = "state_cache_open_write", line = 160)
event = 1108
reentrancy = -31073
}
[13] = {
location = (file = "HttpSM.cc", func = "state_cache_open_write", line = 2400)
event = 1108
reentrancy = 7
}
[14] = {
location = (file = "HttpSM.cc", func = "set_next_state", line = 7407)
event = 34463
reentrancy = 7
}
[15] = {
location = (file = "HttpSM.cc", func = "state_http_server_open", line = 1719)
event = 200
reentrancy = 8
}
[16] = {
location = (file = "HttpSM.cc", func = "do_cache_lookup_and_read", line = 4575)
event = 0
reentrancy = 3
}
[17] = {
location = (file = "HttpSM.cc", func = "state_http_server_open", line = 1719)
event = 101
reentrancy = 1
}
[18] = {
location = (file = "HttpSM.cc", func = "state_send_server_request_header", line = 2007)
event = 103
reentrancy = 1
}
[19] = {
location = (file = "HttpSM.cc", func = "state_read_server_response_header", line = 1849)
event = 100
reentrancy = 1
}
[20] = {
location = (file = "HttpSM.cc", func = "state_api_callback", line = 1337)
event = 60000
reentrancy = 2
}
[21] = {
location = (file = "HttpSM.cc", func = "state_api_callout", line = 1377)
event = 60000
reentrancy = 2
}
[22] = {
location = (file = "HttpSM.cc", func = "setup_server_transfer", line = 6651)
event = 34463
reentrancy = 2
}
[23] = {
location = (file = "HttpSM.cc", func = "tunnel_handler_ua", line = 3195)
event = 3
reentrancy = 0
}
[24] = {
location = (file = "HttpSM.cc", func = "tunnel_handler", line = 2878)
event = 104
reentrancy = 1
}
[25] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[26] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[27] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[28] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[29] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[30] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[31] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[32] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[33] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[34] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[35] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[36] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[37] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[38] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[39] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[40] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[41] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[42] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[43] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[44] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[45] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[46] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[47] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[48] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[49] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[50] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[51] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[52] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[53] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[54] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[55] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[56] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[57] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[58] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[59] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[60] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[61] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[62] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[63] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
[64] = {
location = (file = 0x0000000000000000, func = 0x0000000000000000, line = 0)
event = 0
reentrancy = 0
}
}
history_pos = 25
}
Susan Hinrichs 5:50 PM
This is running ATS 9.0.x plus the hostdb fixes?
randall 5:53 PM
yes, and now “Remove remnants of http/1.1 pipeline logic” (I forgot to revert before running this test)
Susan Hinrichs 6:00 PM
Hmm. My line numbers aren't matching yours. Did you start with ATS/9. 0 and cherry pick in the others?
6:01
Or can you just push your branch?
randall 6:03 PM
I started with 9.x and cherry-picked and then hand applied your last change (too lazy to undo and redo).
6:04
let me clear and re-run
6:04
I also added a warn to verify I actually tried your code (just to make sure I wasn’t losing it)
6:05
and l’ll try with the official upstream 9.0.x (I think we have no cherrypicks so far, but better safe here)
randall 6:45 PM
from https://github.com/randall/trafficserver/tree/6686_9.0.x :
6:45
lldb) print this->history
(History<65>) $0 = {
history = {
[0] = {
location = (file = "HttpSM.cc", func = "state_read_client_request_header", line = 613)
event = 100
reentrancy = 2
}
[1] = {
location = (file = "HttpSM.cc", func = "state_api_callback", line = 1337)
event = 60000
reentrancy = 3
}
[2] = {
location = (file = "HttpSM.cc", func = "state_api_callout", line = 1377)
event = 60000
reentrancy = 3
}
[3] = {
location = (file = "HttpSM.cc", func = "set_next_state", line = 7404)
event = 34463
reentrancy = 3
}
[4] = {
location = (file = "HttpCacheSM.cc", func = "state_cache_open_read", line = 100)
event = 1103
reentrancy = -31073
}
[5] = {
location = (file = "HttpSM.cc", func = "state_cache_open_read", line = 2518)
event = 1103
reentrancy = 4
}
[6] = {
location = (file = "HttpSM.cc", func = "state_api_callback", line = 1337)
event = 60000
reentrancy = 5
}
[7] = {
location = (file = "HttpSM.cc", func = "state_api_callout", line = 1377)
event = 60000
reentrancy = 5
}
[8] = {
location = (file = "HttpSM.cc", func = "state_api_callback", line = 1337)
event = 60000
reentrancy = 6
}
[9] = {
location = (file = "HttpSM.cc", func = "state_api_callout", line = 1377)
event = 60000
reentrancy = 6
}
[10] = {
location = (file = "HttpSM.cc", func = "set_next_state", line = 7364)
event = 34463
reentrancy = 6
}
[11] = {
location = (file = "HttpSM.cc", func = "set_next_state", line = 7513)
event = 34463
reentrancy = 6
}
[12] = {
location = (file = "HttpCacheSM.cc", func = "state_cache_open_write", line = 160)
event = 1108
reentrancy = -31073
}
[13] = {
location = (file = "HttpSM.cc", func = "state_cache_open_write", line = 2400)
event = 1108
reentrancy = 7
}
[14] = {
location = (file = "HttpSM.cc", func = "set_next_state", line = 7410)
event = 34463
reentrancy = 7
}
[15] = {
location = (file = "HttpSM.cc", func = "do_cache_lookup_and_read", line = 4578)
event = 0
reentrancy = 3
}
[16] = {
location = (file = "HttpSM.cc", func = "state_send_server_request_header", line = 2007)
event = 103
reentrancy = 1
}
[17] = {
location = (file = "HttpSM.cc", func = "state_read_server_response_header", line = 1849)
event = 100
reentrancy = 1
}
[18] = {
location = (file = "HttpSM.cc", func = "state_api_callback", line = 1337)
event = 60000
reentrancy = 2
}
[19] = {
location = (file = "HttpSM.cc", func = "state_api_callout", line = 1377)
event = 60000
reentrancy = 2
}
[20] = {
location = (file = "HttpSM.cc", func = "setup_server_transfer", line = 6654)
event = 34463
reentrancy = 2
}
[21] = {
location = (file = "HttpSM.cc", func = "tunnel_handler_ua", line = 3195)
event = 3
reentrancy = 0
}
[22] = {
location = (file = "HttpSM.cc", func = "tunnel_handler", line = 2878)
event = 104
reentrancy = 1
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org