You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "David Brodin (JIRA)" <ji...@apache.org> on 2016/09/03 15:54:20 UTC
[jira] [Created] (TS-4816) ATS 6.2.0 - crashing with broken pipe,
sig11 segmentation fault
David Brodin created TS-4816:
--------------------------------
Summary: ATS 6.2.0 - crashing with broken pipe, sig11 segmentation fault
Key: TS-4816
URL: https://issues.apache.org/jira/browse/TS-4816
Project: Traffic Server
Issue Type: Bug
Components: Manager
Reporter: David Brodin
Hi,
We just upgraded to ATS 6.2.0 via FreeBSD ports:
{noformat}
[root@<machine> ~]# uname -a
FreeBSD <machine> 10.3-RELEASE-p7 FreeBSD 10.3-RELEASE-p7 #0: Thu Aug 11 18:38:15 UTC 2016 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
[root@<machine> ~]# pkg info | grep traff
trafficserver-6.2.0 Fast, scalable and extensible HTTP proxy server
{noformat}
We are experiencing crashes, usually during the day, hardly any during "low" loads, but the mest affecting crashes occur in the early mornings. Along with this we can see a memory leak aswell
We are using ATS as an enterprise proxy to the Internet, and as we have a very good Internet-connection we have also disabled caching.
I'm not sure how I would attach files so here goes :)
manager.log
{noformat}
[Sep 2 11:48:28.017] Manager {0x804006400} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Sep 2 11:48:28.017] Manager {0x804006400} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe)
[Sep 2 11:48:38.305] {0x804006400} STATUS: opened /var/log/trafficserver/manager.log
[Sep 2 11:48:38.305] {0x804006400} NOTE: <DiagsConfig.cc:141 (reconfigure_diags)> updated diags config
[Sep 2 11:48:38.311] Manager {0x804006400} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'FreeBSD' Release: '10.3-RELEASE-p7'
[Sep 2 11:48:38.312] Manager {0x804006400} NOTE: [LocalManager::listenForProxy] Listening on port: 8080 (IPv4)
[Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [LocalManager::listenForProxy] Listening on port: 8080 (IPv6)
[Sep 2 11:48:38.313] Manager {0x804006400} NOTE: [TrafficManager] Setup complete
[Sep 2 11:48:39.321] Manager {0x804006400} NOTE: [LocalManager::startProxy] Launching ts process
[Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '17'
[Sep 2 11:48:39.336] Manager {0x804006400} NOTE: [Alarms::signalAlarm] Server Process born
[Sep 2 11:51:32.574] Manager {0x804006400} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Sep 2 11:51:32.574] Manager {0x804006400} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe)
[Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
[Sep 2 11:51:32.669] Manager {0x804006400} ERROR: [Alarms::signalAlarm] Server Process was reset
[Sep 2 11:51:33.674] Manager {0x804006400} NOTE: [LocalManager::startProxy] Launching ts process
[Sep 2 11:51:33.689] Manager {0x804006400} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
[Sep 2 11:51:33.690] Manager {0x804006400} NOTE: [Alarms::signalAlarm] Server Process born
[Sep 3 04:14:35.380] Manager {0x804006400} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Sep 3 04:14:35.380] Manager {0x804006400} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe)
[Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
[Sep 3 04:14:35.748] Manager {0x804006400} ERROR: [Alarms::signalAlarm] Server Process was reset
[Sep 3 04:14:36.814] Manager {0x804006400} NOTE: [LocalManager::startProxy] Launching ts process
[Sep 3 04:14:36.828] Manager {0x804006400} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
[Sep 3 04:14:36.829] Manager {0x804006400} NOTE: [Alarms::signalAlarm] Server Process born
{noformat}
traffic.out - since this isnt timestamped I'm not sure if I'm leaving some of the stacktrace out:
{noformat}
traffic_server[TrafficManager] ==> Cleaning up and reissuing signal #15
: Terminated
traffic_server: Terminatedtraffic_servertraffic_servertraffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
getpeereid -> 0 (54, Connection reset by peer)[TrafficManager] ==> Cleaning up and reissuing signal #15
traffic_server: Terminated
traffic_server: Terminated
traffic_server: using root directory '/usr/local'
[TrafficManager] ==> signal #15
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Terminated
traffic_server: Terminated
traffic_server: Terminated
traffic_server: Terminated
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
traffic_server: Segmentation fault
traffic_server - STACK TRACE:
0x4af409 <_Z19crash_logger_invokeiP9__siginfoPv+0x69> at /usr/local/bin/traffic_server
0x802735b37 <pthread_sigmask+0x507> at /lib/libthr.so.3
0x80273522c <pthread_getspecific+0xe1c> at /lib/libthr.so.3
{noformat}
/var/log/messages
{noformat}
Sep 2 06:58:03 <machine> traffic_manager[5604]: {0x804006400} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
Sep 2 06:58:03 <machine> kernel: pid 6680 (traffic_server), uid 80: exited on signal 11
Sep 2 06:58:03 <machine> traffic_manager[5604]: {0x804006400} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe)
Sep 2 06:58:04 <machine> traffic_cop[5603]: cannot find traffic_server [1]
Sep 2 06:58:04 <machine> traffic_manager[5604]: {0x804006400} ERROR: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 11: Segmentation fault
Sep 2 06:58:04 <machine> traffic_manager[5604]: {0x804006400} ERROR: [Alarms::signalAlarm] Server Process was reset
Sep 2 06:58:08 <machine> traffic_server[9951]: NOTE: --- traffic_server Starting ---
Sep 2 06:58:08 <machine> traffic_server[9951]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.2.0 - (build # 083112 on Aug 31 2016 at 12:51:58)
Sep 2 06:58:08 <machine> traffic_server[9951]: NOTE: RLIMIT_NOFILE(8):cur(190297),max(190297)
{noformat}
Every other second /var/log/messages is also getting 1-10 lines of this:
{noformat}
Sep 3 17:48:50 <machine> traffic_server[14338]: {0x804008000} ERROR: <HttpSM.cc:1159 (state_raw_http_server_open)> [HttpSM::state_raw_http_server_open] event: EVENT_INTERVAL state: 0 server_entry: 0x0
{noformat}
And a "ps aux" showing mem usage:
{noformat}
[root@<machine> /usr/local/etc/trafficserver]# ps axu | grep "USER\|traff"
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
www 14338 7.9 21.3 1910932 1778200 - S 4:14AM 27:51.59 /usr/local/bin/traffic_server -M --bind_stdout /var/log/trafficserver/traffic.out --bind_stderr /var/log/traff
root 5602 0.0 0.0 14492 2004 - Is Thu10AM 0:00.00 daemon: /usr/local/bin/traffic_cop[5603] (daemon)
root 5603 0.0 0.1 64360 7516 - Ss Thu10AM 0:07.96 /usr/local/bin/traffic_cop
www 10897 0.0 0.2 87544 13492 - S Fri11AM 0:34.78 /usr/local/bin/traffic_manager --bind_stdout /var/log/trafficserver/traffic.out --bind_stderr /var/log/traffic
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)