You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "Leif Hedstrom (JIRA)" <ji...@apache.org> on 2016/01/09 04:28:40 UTC
[jira] [Updated] (TS-3943) ATS 6 crash over and over again on
restart under load
[ https://issues.apache.org/jira/browse/TS-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Leif Hedstrom updated TS-3943:
------------------------------
Fix Version/s: (was: 6.1.0)
6.2.0
> ATS 6 crash over and over again on restart under load
> -----------------------------------------------------
>
> Key: TS-3943
> URL: https://issues.apache.org/jira/browse/TS-3943
> Project: Traffic Server
> Issue Type: Bug
> Components: ICP
> Affects Versions: 6.0.0
> Reporter: Luca Rea
> Assignee: Leif Hedstrom
> Labels: regresion
> Fix For: 6.2.0
>
>
> Hi,
> when ICP is enabled and ATS is under huge load a "service trafficserver stop" followed by a "service trafficserver start" cause a crashing loop (ATS crash, restart, crash again, restart, crash again... )
> {code}
> [E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
> traffic_server: using root directory '/usr/local'
> traffic_server: Segmentation fault (Address not mapped to object [(nil)])traffic_server - STACK TRACE:
> traffic_server: Segmentation fault (Address not mapped to object [(nil)])traffic_server - STACK TRACE:
> /usr/local/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0x99)[0x4a5809]
> /usr/local/bin/traffic_server(_Z19crash_logger_invokeiP7siginfoPv+0x99)[0x4a5809]
> /lib64/libc.so.6[0x31866326a0]
> /usr/local/bin/traffic_server(_ZN12ICPProcessor8ICPQueryEP12ContinuationP3URL+0xb6)[0x4aca16]
> /usr/local/bin/traffic_server(_ZN6HttpSM13do_icp_lookupEv+0x2c)[0x582c7c]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x28e)[0x59c3fe]
> /lib64/libc.so.6[0x31866326a0]
> /usr/local/bin/traffic_server(_ZN12ICPProcessor8ICPQueryEP12ContinuationP3URL+0xb6)[0x4aca16]
> /usr/local/bin/traffic_server(_ZN6HttpSM13do_icp_lookupEv+0x2c)[0x582c7c]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x28e)[0x59c3fe]
> /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
> /usr/local/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x33a)[0x588a4a]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0xbde)[0x59cd4e]
> /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
> /usr/local/bin/traffic_server(_ZN6HttpSM21state_cache_open_readEiPv+0x15c)[0x59a3fc]
> /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x5984f8]
> /usr/local/bin/traffic_server(_ZN11HttpCacheSM21state_cache_open_readEiPv+0x152)[0x5760a2]
> /usr/local/bin/traffic_server(_ZN5Cache9open_readEP12ContinuationPKN3ats10CryptoHashEP7HTTPHdrP21CacheLookupHttpConfig13CacheFragTypePKci+0x2b1)[0x6e26c1]
> /usr/local/bin/traffic_server(_ZN11HttpCacheSM9open_readEPK12HttpCacheKeyP3URLP7HTTPHdrP21CacheLookupHttpConfigl+0xbd)[0x57574d]
> /usr/local/bin/traffic_server(_ZN6HttpSM24do_cache_lookup_and_readEv+0x142)[0x5831d2]
> /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x7b6)[0x59c926]
> /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x229)[0x59c399]
> /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
> /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2ef)[0x595b5f]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
> /usr/local/bin/traffic_server(_ZN6HttpSM32state_read_client_request_headerEiPv+0x3dc)[0x59371c]
> /usr/local/bin/traffic_server(_ZN6HttpSM12main_handlerEiPv+0xd8)[0x5984f8]
> /usr/local/bin/traffic_server(_ZN6HttpSM17handle_api_returnEv+0x2b2)[0x595b22]
> /usr/local/bin/traffic_server(_ZN6HttpSM21attach_client_sessionEP17HttpClientSessionP14IOBufferReader+0x75f)[0x59d8cf]
> /usr/local/bin/traffic_server(_ZN17HttpClientSession15new_transactionEv+0xec)[0x5769dc]
> /usr/local/bin/traffic_server(_ZN18ProxyClientSession14do_api_calloutE12TSHttpHookID+0x14d)[0x4e585d]
> /usr/local/bin/traffic_server(_ZN17HttpClientSession14new_connectionEP14NetVConnectionP9MIOBufferP14IOBufferReaderb+0x224)[0x578654]
> /usr/local/bin/traffic_server(_ZN17HttpSessionAccept6acceptEP14NetVConnectionP9MIOBufferP14IOBufferReader+0x1ea)[0x5729ea]
> /usr/local/bin/traffic_server(_ZN23ProtocolProbeTrampoline17ioCompletionEventEiPv+0x483)[0x4e5343]
> /usr/local/bin/traffic_server[0x71eaf9]
> /usr/local/bin/traffic_server[0x723c0e]
> /usr/local/bin/traffic_server(_ZN10NetHandler12mainNetEventEiP5Event+0x1f2)[0x7151d2]
> /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x125)[0x742d05]
> /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x4c9)[0x743649]
> /usr/local/bin/traffic_server[0x74217a]
> /lib64/libpthread.so.0(+0x3186a079d1)[0x2b864a19f9d1]
> /lib64/libc.so.6(clone+0x6d)[0x31866e88fd]
> /usr/local/bin/traffic_server(_ZN6HttpSM14set_next_stateEv+0x52e)[0x59c69e]
> /usr/local/bin/traffic_server(_ZN6HttpSM16do_hostdb_lookupEv+0x33a)[0x588a4a]
> {code}
> {code}
> Sep 23 16:12:02 redis-01 traffic_manager[22667]: NOTE: --- Manager Starting ---
> Sep 23 16:12:02 redis-01 traffic_manager[22667]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:19:27)
> Sep 23 16:12:06 redis-01 traffic_server[22676]: NOTE: --- traffic_server Starting ---
> Sep 23 16:12:06 redis-01 traffic_server[22676]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:18:47)
> Sep 23 16:12:07 redis-01 kernel: [ET_NET 4][22681]: segfault at 0 ip 00000000004aca16 sp 00002b3bad5ceb00 error 4 in traffic_server[400000+415000]
> Sep 23 16:12:10 redis-01 abrt[22993]: Saved core dump of pid 22676 (/usr/local/bin/traffic_server) to /var/spool/abrt/ccpp-2015-09-23-16:12:07-22676 (913047552 bytes)
> Sep 23 16:12:10 redis-01 abrtd: Directory 'ccpp-2015-09-23-16:12:07-22676' creation detected
> Sep 23 16:12:10 redis-01 abrtd: Executable '/usr/local/bin/traffic_server' doesn't belong to any package and ProcessUnpackaged is set to 'no'
> Sep 23 16:12:10 redis-01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2015-09-23-16:12:07-22676' exited with 1
> Sep 23 16:12:10 redis-01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2015-09-23-16:12:07-22676'
> Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
> Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
> Sep 23 16:12:10 redis-01 traffic_manager[22667]: {0x7fb082e847e0} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe)
> Sep 23 16:12:10 redis-01 traffic_cop[22665]: cop received child status signal [22667 256]
> Sep 23 16:12:10 redis-01 traffic_cop[22665]: traffic_manager not running, making sure traffic_server is dead
> Sep 23 16:12:10 redis-01 traffic_cop[22665]: spawning traffic_manager
> Sep 23 16:12:10 redis-01 traffic_manager[23012]: NOTE: --- Manager Starting ---
> Sep 23 16:12:10 redis-01 traffic_manager[23012]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:19:27)
> Sep 23 16:12:14 redis-01 traffic_server[23024]: NOTE: --- traffic_server Starting ---
> Sep 23 16:12:14 redis-01 traffic_server[23024]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:18:47)
> Sep 23 16:12:15 redis-01 kernel: [ET_NET 255][23280]: segfault at 0 ip 00000000004aca16 sp 00002b7d2aee6b00 error 4 in traffic_server[400000+415000]
> Sep 23 16:12:15 redis-01 abrt[23339]: Not saving repeating crash in '/usr/local/bin/traffic_server'
> Sep 23 16:12:17 redis-01 abrt[23339]: Saved core dump of pid 23024 to core.23024 (906752000 bytes)
> Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
> Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
> Sep 23 16:12:17 redis-01 traffic_manager[23012]: {0x7f81e958e7e0} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe)
> Sep 23 16:12:17 redis-01 traffic_cop[22665]: cop received child status signal [23012 256]
> Sep 23 16:12:17 redis-01 traffic_cop[22665]: traffic_manager not running, making sure traffic_server is dead
> Sep 23 16:12:17 redis-01 traffic_cop[22665]: spawning traffic_manager
> Sep 23 16:12:17 redis-01 traffic_manager[23356]: NOTE: --- Manager Starting ---
> Sep 23 16:12:17 redis-01 traffic_manager[23356]: NOTE: Manager Version: Apache Traffic Server - traffic_manager - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:19:27)
> Sep 23 16:12:21 redis-01 traffic_server[23368]: NOTE: --- traffic_server Starting ---
> Sep 23 16:12:21 redis-01 traffic_server[23368]: NOTE: traffic_server Version: Apache Traffic Server - traffic_server - 6.1.0 - (build # 092315 on Sep 23 2015 at 15:18:47)
> Sep 23 16:12:22 redis-01 kernel: [ET_NET 193][23562]: segfault at 0 ip 00000000004aca16 sp 00002b866aaa2b00 error 4 in traffic_server[400000+415000]
> Sep 23 16:12:22 redis-01 abrt[23684]: Not saving repeating crash in '/usr/local/bin/traffic_server'
> Sep 23 16:12:24 redis-01 abrt[23684]: Saved core dump of pid 23368 to core.23368 (910950400 bytes)
> Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
> Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
> Sep 23 16:12:25 redis-01 traffic_manager[23356]: {0x7fd054c8d7e0} ERROR: <MgmtUtils.cc:289 (mgmt_elog)> (last system error 32: Broken pipe)
> Sep 23 16:12:25 redis-01 traffic_cop[22665]: cop received child status signal [23356 256]
> Sep 23 16:12:25 redis-01 traffic_cop[22665]: traffic_manager not running, making sure traffic_server is dead
> Sep 23 16:12:25 redis-01 traffic_cop[22665]: unable to start traffic_manager, retrying in 60 second(s)
> {code}
> Disabling ICP solve the issue.
> I have never used and tested ICP before (older releases).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)