You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dikang Gu (JIRA)" <ji...@apache.org> on 2015/09/14 21:50:46 UTC
[jira] [Comment Edited] (CASSANDRA-10321) Gossip to dead nodes
caused CPU usage to be 100%
[ https://issues.apache.org/jira/browse/CASSANDRA-10321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744120#comment-14744120 ]
Dikang Gu edited comment on CASSANDRA-10321 at 9/14/15 7:50 PM:
----------------------------------------------------------------
[~mambocab], the cluster is using NetworkTopologyStrategy, and is crossing 3 datacenters.
You can see the dead node in the last line of the log, "2401:db00:2020:716b:face:0:21:0" is the dead node I referred to, and the cassandra process on it is still running, but somehow, the "Thrift active" and "Gossip active" are false in the nodetool info output.
Yeah, we shouldn't expect it caused 100% cpu usage on some nodes, right?
was (Author: dikanggu):
[~mambocab], the cluster is using NetworkTopologyStrategy, and is crossing 3 datacenters.
You can see the dead node in the last line of the log, "2401:db00:2020:716b:face:0:21:0" is the dead node I referred to, and the cassandra process on it is still running, but some now, the "Thrift active" and "Gossip active" are false in the nodetool info output.
Yeah, we shouldn't expect it caused 100% cpu usage on some nodes, right?
> Gossip to dead nodes caused CPU usage to be 100%
> -------------------------------------------------
>
> Key: CASSANDRA-10321
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10321
> Project: Cassandra
> Issue Type: Bug
> Reporter: Dikang Gu
>
> For one node, the cpu usage jumped to 100%, and logs are full of:
> 2015-09-14_16:34:45.56407 WARN 16:34:45 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:46.66616 WARN 16:34:46 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:47.76830 WARN 16:34:47 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:48.87043 WARN 16:34:48 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:49.97253 WARN 16:34:49 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:51.07462 WARN 16:34:51 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:52.17669 WARN 16:34:52 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:53.27880 WARN 16:34:53 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:54.38090 WARN 16:34:54 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:55.48301 WARN 16:34:55 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:56.58509 WARN 16:34:56 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:57.68721 WARN 16:34:57 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:58.78932 WARN 16:34:58 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:34:59.89142 WARN 16:34:59 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:00.99352 WARN 16:35:00 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:02.09563 WARN 16:35:02 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:03.19775 WARN 16:35:03 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:04.29982 WARN 16:35:04 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:05.40187 WARN 16:35:05 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:06.50369 WARN 16:35:06 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:07.60577 WARN 16:35:07 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:08.70779 WARN 16:35:08 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:09.80968 WARN 16:35:09 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:10.91157 WARN 16:35:10 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:12.01365 WARN 16:35:12 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:13.11569 WARN 16:35:13 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:14.21757 WARN 16:35:14 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:15.31942 WARN 16:35:15 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:16.42132 WARN 16:35:16 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:17.52332 WARN 16:35:17 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:18.62511 WARN 16:35:18 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:19.72697 WARN 16:35:19 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:20.82872 WARN 16:35:20 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:21.93074 WARN 16:35:21 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:23.03281 WARN 16:35:23 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:24.13478 WARN 16:35:24 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:25.23672 WARN 16:35:25 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:26.33862 WARN 16:35:26 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:27.44068 WARN 16:35:27 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:28.54267 WARN 16:35:28 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:29.64461 WARN 16:35:29 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:30.74646 WARN 16:35:30 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:35:31.84823 WARN 16:35:31 Gossip stage has 32 pending tasks; skipping status check (no nodes will be marked down)
> 2015-09-14_16:40:09.97371 WARN 16:40:09 /2401:db00:2020:716b:face:0:21:0 has 2311 dropped hints, because node is down past configured hint window.
> This happened to several nodes in our cluster, but one node each time.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)