You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Raji Sankaran <Ra...@vecima.com> on 2021/01/28 05:03:24 UTC
Trafficserver crash analysis
Hi,
We are using ATS 7.1.4 and recently have started noticing trafficserver crashes randomly with the following assertion. We are trying to analyze what could be causing this. A quick google search did not yield promising results. Hence, reaching out for pointers to debug and understand the cause for the crash. Any log/metric that can be enabled which can help in identifying the root cause?
Fatal: HttpTunnel.cc:905: failed assertion `p->num_consumers > 0`
StackTrace:
Nov 17 04:56:48 ftest03 traffic_cop[6416]: spawning traffic_manager
Nov 17 04:56:48 ftest03traffic_manager[6425]: NOTE: --- Manager Starting ---Nov 17 17:33:17 ftest03traffic_server[6848]: Fatal: InkAPI.cc:4865: failed assertion `sdk_sanity_check_mbuffer(*bufp) == TS_SUCCESS`
Nov 17 17:33:21 ftest03 traffic_server[11999]: NOTE: --- traffic_server Starting ---
Nov 17 17:33:21 ftest03 traffic_server[11999]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 7.1.4 - (build # 060114 on Jun 1 2020 at 14:24:51)Nov 19 14:16:41 ftest03 traffic_server[11999]: Fatal: HttpTunnel.cc:905: failed assertion `p->num_consumers > 0`
Nov 19 14:16:44 ftest03 traffic_server[7566]: NOTE: --- traffic_server Starting ---
Nov 19 14:16:44 ftest03 traffic_server[7566]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 7.1.4 - (build # 060114 on Jun 1 2020 at 14:24:51)Nov 22 19:47:50 ftest03 traffic_server[2144]: Fatal: InkAPI.cc:4865: failed assertion `sdk_sanity_check_mbuffer(*bufp) == TS_SUCCESS`
Nov 22 19:47:51 ftest03 traffic_cop[6416]: (test) read failed [104 'Connection reset by peer']
Nov 22 19:47:51 ftest03 traffic_cop[6416]: server heartbeat failed [1]
Nov 22 19:47:54 ftest03 traffic_server[46729]: NOTE: --- traffic_server Starting ---
Nov 22 19:47:54 ftest03 traffic_server[46729]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 7.1.4 - (build # 060114 on Jun 1 2020 at 14:24:51)
Nov 22 19:48:01 ftest03 traffic_cop[6416]: server heartbeat succeeded
Nov 23 05:11:26 ftest03 systemd: Starting Cleanup of Temporary Directories...
Nov 23 05:11:26 ftest03 systemd: Started Cleanup of Temporary Directories.
Traffic.out logs
Fatal: HttpTunnel.cc:905: failed assertion `p->num_consumers > 0`
traffic_server: received signal 6 (Aborted)
traffic_server - STACK TRACE:
/opt/CCURcache/bin/traffic_server(_Z19crash_logger_invokeiP9siginfo_tPv+0x8e)[0x4b063e]
/lib64/libpthread.so.0(+0xf5f0)[0x2b65110585f0]
/lib64/libc.so.6(gsignal+0x37)[0x2b6511cbe337]
/lib64/libc.so.6(abort+0x148)[0x2b6511cbfa28]
/opt/CCURcache/lib64/libtsutil.so.7(+0x2debb)[0x2b650f739ebb]
/opt/CCURcache/lib64/libtsutil.so.7(+0x2bf45)[0x2b650f737f45]
/opt/CCURcache/bin/traffic_server(_ZN10HttpTunnel12producer_runEP18HttpTunnelProducer+0x10b6)[0x5ccfc6]
/opt/CCURcache/bin/traffic_server(_ZN10HttpTunnel10tunnel_runEP18HttpTunnelProducer+0x29)[0x5cd0f9]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x416)[0x591b96]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM32call_transact_and_set_next_stateEPFvPN12HttpTransact5StateEE+0x32)[0x57f2a2]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x8a)[0x5913ba]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x43e)[0x5896be]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x135)[0x590ca5]
/opt/CCURcache/bin/traffic_server(TSHttpTxnReenable+0x249)[0x4da759]
/opt/CCURcache/libexec/trafficserver/collapsed_forwarding.so(+0x1595)[0x2b6644240595]
/opt/CCURcache/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x8e)[0x4c5c0e]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x234)[0x5894b4]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x135)[0x590ca5]
/opt/CCURcache/bin/traffic_server(TSHttpTxnReenable+0x249)[0x4da759]
/opt/CCURcache/libexec/trafficserver/RouterPlugin.so(_ZN12RouterPlugin25handleReadResponseHeadersERN9atscppapi11TransactionE+0x60a)[0x2b6648a9ed26]
/opt/CCURcache/lib64/libatscppapi.so.7(+0x1300d)[0x2b664467400d]
/opt/CCURcache/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x8e)[0x4c5c0e]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x234)[0x5894b4]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM18state_api_callbackEiPv+0x135)[0x590ca5]
/opt/CCURcache/bin/traffic_server(TSHttpTxnReenable+0x249)[0x4da759]
/opt/CCURcache/lib64/libatscppapi.so.7(+0x1f305)[0x2b6644680305]
/opt/CCURcache/bin/traffic_server(_ZN15INKContInternal12handle_eventEiPv+0x8e)[0x4c5c0e]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM17state_api_calloutEiPv+0x234)[0x5894b4]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM33state_read_server_response_headerEiPv+0x350)[0x58aa10]
/opt/CCURcache/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xbd)[0x590e5d]
/opt/CCURcache/bin/traffic_server[0x73298e]
/opt/CCURcache/bin/traffic_server(_ZN10NetHandler15waitForActivityEl+0x2c0)[0x7223c0]
/opt/CCURcache/bin/traffic_server(_ZN7EThread15execute_regularEv+0x3f2)[0x7552b2]
/opt/CCURcache/bin/traffic_server[0x753fda]
/lib64/libpthread.so.0(+0x7e65)[0x2b6511050e65]
/lib64/libc.so.6(clone+0x6d)[0x2b6511d8688d]
Thanks in advance for the help.
Raji