You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@trafficserver.apache.org by GitBox <gi...@apache.org> on 2020/11/20 01:56:26 UTC
[GitHub] [trafficserver] shinrich opened a new pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
shinrich opened a new pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336
The issue describes most of the issue. This error came in with PR #7237. We were internally running the first version of that PR, but in reviewing the PR @masaori335 found a ink_assert being triggered. I over fixed the change, so when we upgraded our build to run the final version of the PR, we started seeing many crashes due to lingering continuations being associated with VC vio's.
This PR fixed the issue be not proactivity marking the producer dead. So the normal cleanup paths will still be executed.
This closes #7333
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
shinrich commented on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-734017972
Are you seeing this assert from running autest?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] zwoop commented on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
zwoop commented on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-740163792
Cherry-picked to v9.0.x branch.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich edited a comment on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
shinrich edited a comment on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-734017972
Are you seeing this assert from running autest? Does seem like it would be encountered pretty easily, but didn't run into that one. Will set that up again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 commented on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
masaori335 commented on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-737597557
[approve ci autest]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
shinrich commented on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-735923341
Was able to reproduce the assert as well. I have a possible couple fixes. Will have to wait until tomorrow to try in our production environment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 commented on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
masaori335 commented on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-734061780
> Are you seeing this assert from running autest? Does seem like it would be encountered pretty easily, but didn't run into that one. Will set that up again.
Not AuTest. I saw this with h2spec + patch on the #7237.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 commented on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
masaori335 commented on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-733398763
Just comment about the design of HttpTunnel, relying on these flags (`active`, `call_sm`, `alive`, ...) seems pretty fragile.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 commented on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
masaori335 commented on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-733388875
```
[Nov 25 09:47:45.783] [ET_NET 0] DEBUG: <HttpTunnel.cc:1307 (consumer_handler)> (http_tunnel) [0] consumer_handler [http server post VC_EVENT_WRITE_COMPLETE/TS_EVENT_VCONN_WRITE_COMPLETE]
[Nov 25 09:47:45.783] [ET_NET 0] DEBUG: <HttpSM.cc:3625 (tunnel_handler_post_server)> (http) [0] [&HttpSM::tunnel_handler_post_server, VC_EVENT_WRITE_COMPLETE/TS_EVENT_VCONN_WRITE_COMPLETE]
[Nov 25 09:47:45.783] [ET_NET 0] DEBUG: <HttpSM.cc:2639 (main_handler)> (http) [0] [HttpSM::main_handler, HTTP_TUNNEL_EVENT_DONE]
[Nov 25 09:47:45.783] [ET_NET 0] DEBUG: <HttpSM.cc:2740 (tunnel_handler_post)> (http) [0] [&HttpSM::tunnel_handler_post, HTTP_TUNNEL_EVENT_DONE]
Fatal: HttpTunnel.cc:473: failed assertion `producer.alive == false`
traffic_server: received signal 6 (Abort trap: 6)
traffic_server - STACK TRACE:
0 traffic_server 0x000000010fbe5c28 _Z19crash_logger_invokeiP9__siginfoPv + 136
1 libsystem_platform.dylib 0x00007fff6e4375fd _sigtramp + 29
2 ??? 0x0000000000000000 0x0 + 0
3 libsystem_c.dylib 0x00007fff6e30d808 abort + 120
4 libtscore.10.dylib 0x0000000110a6e04e _Z9ink_abortPKcz + 350
5 libtscore.10.dylib 0x0000000110a6b16c _ink_assert + 44
6 traffic_server 0x000000010fd2c728 _ZN10HttpTunnel5resetEv + 216
7 traffic_server 0x000000010fcc9dad _ZN6HttpSM26tunnel_handler_post_or_putEP18HttpTunnelProducer + 797
8 traffic_server 0x000000010fcb738f _ZN6HttpSM19tunnel_handler_postEiPv + 1135
9 traffic_server 0x000000010fcb221a _ZN6HttpSM12main_handlerEiPv + 1034
10 traffic_server 0x000000010fbe909f _ZN12Continuation11handleEventEiPv + 255
11 traffic_server 0x000000010fd2c60f _ZN10HttpTunnel12main_handlerEiPv + 463
12 traffic_server 0x000000010fbe909f _ZN12Continuation11handleEventEiPv + 255
13 traffic_server 0x000000010ffae3b5 _ZL23write_signal_and_updateiP18UnixNetVConnection + 165
14 traffic_server 0x000000010ffae6ad _ZL17write_signal_doneiP10NetHandlerP18UnixNetVConnection + 45
15 traffic_server 0x000000010ffadd75 _Z15write_to_net_ioP10NetHandlerP18UnixNetVConnectionP7EThread + 3205
16 traffic_server 0x000000010ffad0e3 _Z12write_to_netP10NetHandlerP18UnixNetVConnectionP7EThread + 131
17 traffic_server 0x000000010ffb1875 _ZN18UnixNetVConnection12net_write_ioEP10NetHandlerP7EThread + 37
18 traffic_server 0x000000010ffa09ba _ZN10NetHandler18process_ready_listEv + 394
19 traffic_server 0x000000010ffa1326 _ZN10NetHandler15waitForActivityEx + 2166
20 traffic_server 0x000000010ffe2771 _ZN7EThread15execute_regularEv + 1025
21 traffic_server 0x000000010ffe2bed _ZN7EThread7executeEv + 285
22 traffic_server 0x000000010ffe0dab _ZL21spawn_thread_internalPv + 107
23 libsystem_pthread.dylib 0x00007fff6e443109 _pthread_start + 148
24 libsystem_pthread.dylib 0x00007fff6e43eb8b thread_start + 15
zsh: abort /opt/ats-review/bin/traffic_server -T http
```
The assert which we faced at #7237 is back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] masaori335 edited a comment on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
masaori335 edited a comment on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-734061780
> Are you seeing this assert from running autest? Does seem like it would be encountered pretty easily, but didn't run into that one. Will set that up again.
Not AuTest. I saw this with h2spec + patch on the #7237. I was trying to make sure this change doesn't break fixes of #7237.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich commented on pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
shinrich commented on pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336#issuecomment-737375285
Pushed up another commit. This should avoid the debug assert, and has run without incident for a couple hours in production. Will continue to keep an eye on that build today.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficserver] shinrich merged pull request #7336: Adjust flags to ensure tunnel producer is cleaned up
Posted by GitBox <gi...@apache.org>.
shinrich merged pull request #7336:
URL: https://github.com/apache/trafficserver/pull/7336
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org