You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "Zhao Yongming (JIRA)" <ji...@apache.org> on 2012/10/17 11:44:03 UTC
[jira] [Assigned] (TS-1346) syn cookie may crash config update in
cluster
[ https://issues.apache.org/jira/browse/TS-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhao Yongming reassigned TS-1346:
---------------------------------
Assignee: kuotai
> syn cookie may crash config update in cluster
> ---------------------------------------------
>
> Key: TS-1346
> URL: https://issues.apache.org/jira/browse/TS-1346
> Project: Traffic Server
> Issue Type: Bug
> Components: Clustering, Management
> Affects Versions: 3.3.1, 3.0.5
> Reporter: Zhao Yongming
> Assignee: kuotai
> Fix For: 3.3.2
>
>
> in clustering, we have about 48 boxes, where the config sync and others may trigger the 8088 syn cookie or so. which may case the manager killed:
> {code}
> [179378.192955] possible SYN flooding on port 8088. Sending cookies.
> [179472.331827] possible SYN flooding on port 8088. Sending cookies.
> [235126.729593] possible SYN flooding on port 8088. Sending cookies.
> [235335.369386] possible SYN flooding on port 8088. Sending cookies.
> [235483.128214] possible SYN flooding on port 8088. Sending cookies.
> [235639.853147] possible SYN flooding on port 8088. Sending cookies.
> [235785.611114] possible SYN flooding on port 8088. Sending cookies.
> [235852.501364] possible SYN flooding on port 8088. Sending cookies.
> [238275.390711] possible SYN flooding on port 8088. Sending cookies.
> {code}
> {code}
> [Jul 12 11:37:15.254] Manager {0x7fe8b33d5700} NOTE: [drainIncomingChannel] file req: records.config v: 8963 bytes: 29295
> [Jul 12 11:37:15.301] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8959
> [Jul 12 11:37:15.301] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:37:15.314] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8960
> [Jul 12 11:37:15.315] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:37:27.633] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8963 n: 8964
> [Jul 12 11:37:27.892] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8964 n: 8965
> [Jul 12 11:37:32.374] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8965 n: 8966
> [Jul 12 11:37:39.608] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8965
> [Jul 12 11:37:39.608] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:37:49.975] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8966 n: 8967
> [Jul 12 11:37:50.067] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8967 n: 8968
> [Jul 12 11:38:10.051] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8968 n: 8969
> [Jul 12 11:38:39.222] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8969 n: 8970
> [Jul 12 11:38:40.194] Manager {0x7fe8b33d5700} NOTE: marking server on node 222.186.49.169 as up
> [Jul 12 11:38:49.250] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:49.250] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:49.250] Manager {0x7fe8b33d5700} NOTE: marking server on node 222.186.49.165 as up
> [Jul 12 11:38:49.252] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:49.252] Manager {0x7fe8b4a6f7e0} NOTE: marking server on node 222.186.49.185 as down
> [Jul 12 11:38:49.252] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:49.252] Manager {0x7fe8b33d5700} NOTE: marking server on node 222.186.49.185 as up
> [Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:49.253] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.491] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.491] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.492] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.492] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.493] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.493] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.494] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.495] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.495] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.496] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.496] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.497] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.497] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.497] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.498] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.498] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.498] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.499] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.499] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: (last system error 4: Interrupted system call)
> [Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::sendReliableMessageReadTillClose] Unable to connect
> [Jul 12 11:38:54.500] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:38:54.501] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.501] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:38:54.502] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.502] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:38:54.503] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.503] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:38:54.503] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.504] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:38:54.504] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.504] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:38:54.505] Manager {0x7fe8b33d5700} ERROR: [ClusterCom::handleMultiCastFilePacket] Update failed
> [Jul 12 11:38:54.505] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:38:54.624] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8970 n: 8973
> [Jul 12 11:39:02.360] Manager {0x7fe8b4a6f7e0} NOTE: marking manager on node 222.186.49.196 as down
> [Jul 12 11:39:02.360] Manager {0x7fe8b4a6f7e0} NOTE: marking server on node 222.186.49.196 as down
> [Jul 12 11:39:02.388] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8973 n: 8974
> [Jul 12 11:39:02.389] Manager {0x7fe8b33d5700} NOTE: marking manager on node 222.186.49.196 as up
> [Jul 12 11:39:03.279] Manager {0x7fe8b33d5700} NOTE: marking server on node 222.186.49.196 as up
> [Jul 12 11:39:05.471] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8974 n: 8975
> [Jul 12 11:39:18.916] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8975 n: 8976
> [Jul 12 11:39:18.918] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8975
> [Jul 12 11:39:18.918] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:18.999] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8976 n: 8977
> [Jul 12 11:39:22.973] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8977 n: 8978
> [Jul 12 11:39:28.455] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8978 n: 8979
> [Jul 12 11:39:29.013] Manager {0x7fe8b33d5700} NOTE: [drainIncomingChannel] file req: records.config v: 8979 bytes: 29295
> [Jul 12 11:39:32.995] Manager {0x7fe8b33d5700} NOTE: [ClusterCom::handleMultiCastFilePacket] Updated 'records.config' o: 8979 n: 8980
> [Jul 12 11:39:32.997] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.998] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:32.999] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:37.400] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:37.400] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:37.403] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:37.403] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:37.403] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:37.404] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [Jul 12 11:39:37.404] Manager {0x7fe8b33d5700} ERROR: [drainIncomingChannel] Error file req: records.config ver: 8979
> [Jul 12 11:39:37.404] Manager {0x7fe8b33d5700} ERROR: (last system error 111: Connection refused)
> [TrafficManager] ==> Cleaning up and reissuing signal #11
> [Jul 12 11:39:48.329] Manager {0x7fe8b33d5700} ERROR: [TrafficManager] ==> Cleaning up and reissuing signal #11
> [Jul 12 11:39:48.329] Manager {0x7fe8b33d5700} ERROR: (last system error 104: Connection reset by peer)
> [E. Mgmt] log ==> [TrafficManager] using root directory '/usr'
> [Jul 12 11:39:49.346] {0x7fc772ec37e0} STATUS: opened /var/log/trafficserver/manager.log
> [Jul 12 11:39:49.346] {0x7fc772ec37e0} NOTE: updated diags config
> [Jul 12 11:39:49.353] Manager {0x7fc772ec37e0} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '2.6.32-220.17.1.tb619.el6.x86_6'
> [Jul 12 11:39:49.356] Manager {0x7fc772ec37e0} NOTE: [LocalManager::listenForProxy] Listening on port: 8080
> [Jul 12 11:39:49.357] Manager {0x7fc772ec37e0} NOTE: [LocalManager::listenForProxy] Listening on port: 81
> [Jul 12 11:39:49.357] Manager {0x7fc772ec37e0} NOTE: [TrafficManager] Setup complete
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira