You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficserver.apache.org by Brendan Webb <we...@ohsu.edu> on 2014/06/11 20:10:18 UTC
ATS 4.2.1 crashing
Back in April, I wrote about ATS 4.2.0 crashing after an upgrade from
3.0.4. Alan Carroll mentioned it might be related to TS-2564. I've since
upgraded to ATS 4.2.1 and I'm still seeing similar crashes. Any ideas?
traffic.out
[TrafficServer] using root directory '/app00/trafficserver'
FATAL: InkAPI.cc:2669: failed assert `sdk_sanity_check_mbuffer(bufp) ==
TS_SUCCESS`
/app00/trafficserver/bin/traffic_server - STACK TRACE:
/app00/trafficserver/lib/libtsutil.so.4(+0x1f228)[0x2b92fcf54228]
/app00/trafficserver/lib/libtsutil.so.4(+0x1d79f)[0x2b92fcf5279f]
/app00/trafficserver/bin/traffic_server(TSMimeHdrFieldFind+0x35)[0x4b3845]
/app00/trafficserver/libexec/trafficserver/rfc5861.so(+0x32c8)[0x2b93043062
c8]
/app00/trafficserver/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi
+0x8f)[0x6ab26f]
/app00/trafficserver/bin/traffic_server(_ZN7EThread7executeEv+0x61b)[0x6abd
9b]
/app00/trafficserver/bin/traffic_server[0x6aa5fa]
/lib64/libpthread.so.0[0x3a738077e1]
/lib64/libc.so.6(clone+0x6d)[0x3a734e153d]
manager.log
[Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
[Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} NOTE:
[LocalManager::mgmtShutdown] Executing shutdown request.
[Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} NOTE:
[LocalManager::processShutdown] Executing process shutdown request.
[Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
[Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} ERROR: (last system error
32: Broken pipe)
[Jun 10 08:38:33.851] {0x7fab56b727e0} STATUS: opened
/app00/trafficserver/var/log/trafficserver/manager.log
[Jun 10 08:38:33.851] {0x7fab56b727e0} NOTE: updated diags config
[Jun 10 08:38:33.876] Manager {0x7fab56b727e0} NOTE:
[ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '2.6.32
-71.el6.x86_64'
[Jun 10 08:38:33.906] Manager {0x7fab56b727e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 80
[Jun 10 08:38:33.906] Manager {0x7fab56b727e0} NOTE:
[LocalManager::listenForProxy] Listening on port: 443
[Jun 10 08:38:33.906] Manager {0x7fab56b727e0} NOTE: [TrafficManager]
Setup complete
[Jun 10 08:38:34.918] Manager {0x7fab56b727e0} NOTE:
[LocalManager::startProxy] Launching ts process
[Jun 10 08:38:34.927] Manager {0x7fab56b727e0} NOTE:
[LocalManager::pollMgmtProcessServer] New process connecting fd '14'
[Jun 10 08:38:34.927] Manager {0x7fab56b727e0} NOTE: [Alarms::signalAlarm]
Server Process born
On 4/14/14 2:04 PM, "Brendan Webb" <we...@ohsu.edu> wrote:
>After upgrading from 3.0.4 to 4.2.0, ATS is crashing quite frequently on
>me. Anyone seeing similar crashes?
>
>traffic.out
>[E. Mgmt] log ==> [TrafficManager] using root directory
>'/app00/trafficserver'
>[TrafficServer] using root directory '/app00/trafficserver'
>FATAL: InkAPI.cc:2669: failed assert `sdk_sanity_check_mbuffer(bufp) ==
>TS_SUCCESS`
>/app00/trafficserver/bin/traffic_server - STACK TRACE:
>/app00/trafficserver/lib/libtsutil.so.4(+0x1f228)[0x2b5677816228]
>/app00/trafficserver/lib/libtsutil.so.4(+0x1d79f)[0x2b567781479f]
>/app00/trafficserver/bin/traffic_server(TSMimeHdrFieldFind+0x35)[0x4b36f5]
>/app00/trafficserver/libexec/trafficserver/rfc5861.so(+0x32c8)[0x2b567edea
>2c8]
>/app00/trafficserver/bin/traffic_server(_ZN7EThread13process_eventEP5Event
>i+0x8f)[0x6aad2f]
>/app00/trafficserver/bin/traffic_server(_ZN7EThread7executeEv+0x61b)[0x6ab
>85b]
>/app00/trafficserver/bin/traffic_server[0x6aa0ba]
>/lib64/libpthread.so.0[0x3acc4077e1]
>/lib64/libc.so.6(clone+0x6d)[0x3acc0e153d]
>
>manager.log
>[Apr 14 13:02:24.119] {0x7f997f82c7e0} STATUS: opened
>/app00/trafficserver/var/log/trafficserver/manager.log
>[Apr 14 13:02:24.119] {0x7f997f82c7e0} NOTE: updated diags config
>[Apr 14 13:02:24.122] Manager {0x7f997f82c7e0} NOTE: [Rollback::openFile]
>Open of prefetch.config failed: Permission denied
>[Apr 14 13:02:24.122] Manager {0x7f997f82c7e0} NOTE: [Rollback::Rollback]
>Config file is read-only : prefetch.config
>[Apr 14 13:02:24.132] Manager {0x7f997f82c7e0} NOTE:
>[ClusterCom::ClusterCom] Node running on OS: 'Linux' Release:
>'2.6.32-71.el6.x86_64'
>[Apr 14 13:02:24.132] Manager {0x7f997f82c7e0} NOTE:
>[LocalManager::listenForProxy] Listening on port: 80
>[Apr 14 13:02:24.132] Manager {0x7f997f82c7e0} NOTE:
>[LocalManager::listenForProxy] Listening on port: 443
>[Apr 14 13:02:24.132] Manager {0x7f997f82c7e0} NOTE: [TrafficManager]
>Setup complete
>[Apr 14 13:02:25.158] Manager {0x7f997f82c7e0} NOTE:
>[LocalManager::startProxy] Launching ts process
>[Apr 14 13:02:25.174] Manager {0x7f997f82c7e0} NOTE:
>[LocalManager::pollMgmtProcessServer] New process connecting fd '14'
>[Apr 14 13:02:25.174] Manager {0x7f997f82c7e0} NOTE:
>[Alarms::signalAlarm] Server Process born
>[Apr 14 13:41:05.556] Manager {0x7f997f82c7e0} FATAL:
>[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
>[Apr 14 13:41:05.556] Manager {0x7f997f82c7e0} NOTE:
>[LocalManager::mgmtShutdown] Executing shutdown request.
>[Apr 14 13:41:05.556] Manager {0x7f997f82c7e0} NOTE:
>[LocalManager::processShutdown] Executing process shutdown request.
>[Apr 14 13:41:05.557] Manager {0x7f997f82c7e0} ERROR:
>[LocalManager::sendMgmtMsgToProcesses] Error writing message
>[Apr 14 13:41:05.558] Manager {0x7f997f82c7e0} ERROR: (last system error
>32: Broken pipe)
>
>diags.log
>[Apr 14 13:41:11.420] Server {0x2b0a8d249710} ERROR:
>[SSL_NetVConnection::ssl_read_from_net] SSL_ERROR_SYSCALL, underlying IO
>error: Connection reset by peer
>
Re: ATS 4.2.1 crashing
Posted by James Peach <jp...@apache.org>.
On Jun 11, 2014, at 11:10 AM, Brendan Webb <we...@ohsu.edu> wrote:
> Back in April, I wrote about ATS 4.2.0 crashing after an upgrade from
> 3.0.4. Alan Carroll mentioned it might be related to TS-2564. I've since
> upgraded to ATS 4.2.1 and I'm still seeing similar crashes. Any ideas?
this looks like https://issues.apache.org/jira/browse/TS-2749
>
> traffic.out
> [TrafficServer] using root directory '/app00/trafficserver'
> FATAL: InkAPI.cc:2669: failed assert `sdk_sanity_check_mbuffer(bufp) ==
> TS_SUCCESS`
> /app00/trafficserver/bin/traffic_server - STACK TRACE:
> /app00/trafficserver/lib/libtsutil.so.4(+0x1f228)[0x2b92fcf54228]
> /app00/trafficserver/lib/libtsutil.so.4(+0x1d79f)[0x2b92fcf5279f]
> /app00/trafficserver/bin/traffic_server(TSMimeHdrFieldFind+0x35)[0x4b3845]
> /app00/trafficserver/libexec/trafficserver/rfc5861.so(+0x32c8)[0x2b93043062
> c8]
> /app00/trafficserver/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi
> +0x8f)[0x6ab26f]
> /app00/trafficserver/bin/traffic_server(_ZN7EThread7executeEv+0x61b)[0x6abd
> 9b]
> /app00/trafficserver/bin/traffic_server[0x6aa5fa]
> /lib64/libpthread.so.0[0x3a738077e1]
> /lib64/libc.so.6(clone+0x6d)[0x3a734e153d]
>
>
> manager.log
> [Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} FATAL:
> [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
> [Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} NOTE:
> [LocalManager::mgmtShutdown] Executing shutdown request.
> [Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} NOTE:
> [LocalManager::processShutdown] Executing process shutdown request.
> [Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} ERROR:
> [LocalManager::sendMgmtMsgToProcesses] Error writing message
> [Jun 10 08:38:33.829] Manager {0x7f00deaca7e0} ERROR: (last system error
> 32: Broken pipe)
> [Jun 10 08:38:33.851] {0x7fab56b727e0} STATUS: opened
> /app00/trafficserver/var/log/trafficserver/manager.log
> [Jun 10 08:38:33.851] {0x7fab56b727e0} NOTE: updated diags config
> [Jun 10 08:38:33.876] Manager {0x7fab56b727e0} NOTE:
> [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '2.6.32
> -71.el6.x86_64'
> [Jun 10 08:38:33.906] Manager {0x7fab56b727e0} NOTE:
> [LocalManager::listenForProxy] Listening on port: 80
> [Jun 10 08:38:33.906] Manager {0x7fab56b727e0} NOTE:
> [LocalManager::listenForProxy] Listening on port: 443
> [Jun 10 08:38:33.906] Manager {0x7fab56b727e0} NOTE: [TrafficManager]
> Setup complete
> [Jun 10 08:38:34.918] Manager {0x7fab56b727e0} NOTE:
> [LocalManager::startProxy] Launching ts process
> [Jun 10 08:38:34.927] Manager {0x7fab56b727e0} NOTE:
> [LocalManager::pollMgmtProcessServer] New process connecting fd '14'
> [Jun 10 08:38:34.927] Manager {0x7fab56b727e0} NOTE: [Alarms::signalAlarm]
> Server Process born
>
>
>
>
> On 4/14/14 2:04 PM, "Brendan Webb" <we...@ohsu.edu> wrote:
>
>> After upgrading from 3.0.4 to 4.2.0, ATS is crashing quite frequently on
>> me. Anyone seeing similar crashes?
>>
>> traffic.out
>> [E. Mgmt] log ==> [TrafficManager] using root directory
>> '/app00/trafficserver'
>> [TrafficServer] using root directory '/app00/trafficserver'
>> FATAL: InkAPI.cc:2669: failed assert `sdk_sanity_check_mbuffer(bufp) ==
>> TS_SUCCESS`
>> /app00/trafficserver/bin/traffic_server - STACK TRACE:
>> /app00/trafficserver/lib/libtsutil.so.4(+0x1f228)[0x2b5677816228]
>> /app00/trafficserver/lib/libtsutil.so.4(+0x1d79f)[0x2b567781479f]
>> /app00/trafficserver/bin/traffic_server(TSMimeHdrFieldFind+0x35)[0x4b36f5]
>> /app00/trafficserver/libexec/trafficserver/rfc5861.so(+0x32c8)[0x2b567edea
>> 2c8]
>> /app00/trafficserver/bin/traffic_server(_ZN7EThread13process_eventEP5Event
>> i+0x8f)[0x6aad2f]
>> /app00/trafficserver/bin/traffic_server(_ZN7EThread7executeEv+0x61b)[0x6ab
>> 85b]
>> /app00/trafficserver/bin/traffic_server[0x6aa0ba]
>> /lib64/libpthread.so.0[0x3acc4077e1]
>> /lib64/libc.so.6(clone+0x6d)[0x3acc0e153d]
>>
>> manager.log
>> [Apr 14 13:02:24.119] {0x7f997f82c7e0} STATUS: opened
>> /app00/trafficserver/var/log/trafficserver/manager.log
>> [Apr 14 13:02:24.119] {0x7f997f82c7e0} NOTE: updated diags config
>> [Apr 14 13:02:24.122] Manager {0x7f997f82c7e0} NOTE: [Rollback::openFile]
>> Open of prefetch.config failed: Permission denied
>> [Apr 14 13:02:24.122] Manager {0x7f997f82c7e0} NOTE: [Rollback::Rollback]
>> Config file is read-only : prefetch.config
>> [Apr 14 13:02:24.132] Manager {0x7f997f82c7e0} NOTE:
>> [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release:
>> '2.6.32-71.el6.x86_64'
>> [Apr 14 13:02:24.132] Manager {0x7f997f82c7e0} NOTE:
>> [LocalManager::listenForProxy] Listening on port: 80
>> [Apr 14 13:02:24.132] Manager {0x7f997f82c7e0} NOTE:
>> [LocalManager::listenForProxy] Listening on port: 443
>> [Apr 14 13:02:24.132] Manager {0x7f997f82c7e0} NOTE: [TrafficManager]
>> Setup complete
>> [Apr 14 13:02:25.158] Manager {0x7f997f82c7e0} NOTE:
>> [LocalManager::startProxy] Launching ts process
>> [Apr 14 13:02:25.174] Manager {0x7f997f82c7e0} NOTE:
>> [LocalManager::pollMgmtProcessServer] New process connecting fd '14'
>> [Apr 14 13:02:25.174] Manager {0x7f997f82c7e0} NOTE:
>> [Alarms::signalAlarm] Server Process born
>> [Apr 14 13:41:05.556] Manager {0x7f997f82c7e0} FATAL:
>> [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
>> [Apr 14 13:41:05.556] Manager {0x7f997f82c7e0} NOTE:
>> [LocalManager::mgmtShutdown] Executing shutdown request.
>> [Apr 14 13:41:05.556] Manager {0x7f997f82c7e0} NOTE:
>> [LocalManager::processShutdown] Executing process shutdown request.
>> [Apr 14 13:41:05.557] Manager {0x7f997f82c7e0} ERROR:
>> [LocalManager::sendMgmtMsgToProcesses] Error writing message
>> [Apr 14 13:41:05.558] Manager {0x7f997f82c7e0} ERROR: (last system error
>> 32: Broken pipe)
>>
>> diags.log
>> [Apr 14 13:41:11.420] Server {0x2b0a8d249710} ERROR:
>> [SSL_NetVConnection::ssl_read_from_net] SSL_ERROR_SYSCALL, underlying IO
>> error: Connection reset by peer
>>
>