Gossip Stage ERROR

[Aiman Parvaiz <ai...@flipagram.com>]

We are running C* 2.0.15, recently 2 of our 10 nodes had to be forcefully
removed. Cluster is behaving fine since then as we are not seeing any
issues with production except that nodes every now and then throw out the
following error:


Jun 19 17:18:35 cass-prod5.localdomain cassandra: 2015-06-19 17:18:35,091
INFO GossipStage:37 Gossiper.markDead - InetAddress /10.198.144.205 is now
DOWN
Jun 19 17:18:35 cass-prod5.localdomain cassandra: 2015-06-19 17:18:35,093
INFO GossipStage:37 StorageService.excise - Removing tokens
[5854867837086423915] for /10.198.144.205
Jun 19 17:18:56 cass-prod5.localdomain cassandra: 2015-06-19 17:18:56,307
INFO GossipStage:37 Gossiper.markDead - InetAddress /10.238.2.149 is now
DOWN
Jun 19 17:18:56 cass-prod5.localdomain cassandra: 2015-06-19 17:18:56,308
INFO GossipStage:37 StorageService.excise - Removing tokens
[7296642456956339369] for /10.238.2.149
Jun 19 17:19:41 cass-prod5.localdomain cassandra: 2015-06-19 17:19:41,108
INFO GossipStage:37 Gossiper.markDead - InetAddress /10.198.144.205 is now
DOWN
Jun 19 17:19:41 cass-prod5.localdomain cassandra: 2015-06-19 17:19:41,109
INFO GossipStage:37 StorageService.excise - Removing tokens
[5854867837086423915] for /10.198.144.205
Jun 19 17:20:00 cass-prod5.localdomain cassandra: 2015-06-19 17:20:00,322
INFO GossipStage:37 Gossiper.markDead - InetAddress /10.238.2.149 is now
DOWN
Jun 19 17:20:00 cass-prod5.localdomain cassandra: 2015-06-19 17:20:00,324
INFO GossipStage:37 StorageService.excise - Removing tokens
[7296642456956339369] for /10.238.2.149
Jun 19 17:20:00 cass-prod5.localdomain cassandra: 2015-06-19 17:20:00,329
ERROR GossipStage:37 CassandraDaemon.uncaughtException - Exception in
thread Thread[GossipStage:37,5,main]
Jun 19 17:20:00 cass-prod5.localdomain java.lang.NullPointerException
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.service.StorageService.getRpcaddress(StorageService.java:1067)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.transport.Server$EventNotifier.getRpcAddress(Server.java:345)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.transport.Server$EventNotifier.onLeaveCluster(Server.java:366)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.service.StorageService.excise(StorageService.java:1790)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.service.StorageService.excise(StorageService.java:1798)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.service.StorageService.handleStateLeft(StorageService.java:1701)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.service.StorageService.onChange(StorageService.java:1361)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.service.StorageService.onJoin(StorageService.java:1995)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1003)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1102)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49)
Jun 19 17:20:00 cass-prod5.localdomain     at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62)
Jun 19 17:20:00 cass-prod5.localdomain     at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Jun 19 17:20:00 cass-prod5.localdomain     at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Jun 19 17:20:00 cass-prod5.localdomain     at
java.lang.Thread.run(Thread.java:745)




Each node says the same thing for the same 2 IPs. These IPs belong to the
nodes we had to force remove. I have checked the systems.peers and no node
is showing these 2 as their peers. Nodetool status is also showing the
right boxes in ring. Not sure where these IPs are coming from in the
cluster. Can someone throw any light on this please.

Thanks