You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dikang Gu (JIRA)" <ji...@apache.org> on 2016/05/25 22:00:14 UTC
[jira] [Updated] (CASSANDRA-11892) Can not replace a dead host
[ https://issues.apache.org/jira/browse/CASSANDRA-11892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dikang Gu updated CASSANDRA-11892:
----------------------------------
Description:
I got some errors when trying to replace a dead host.
{code}
2016-05-25_20:59:37.61838 ERROR 20:59:37 [main]: Exception encountered during startup
2016-05-25_20:59:37.61839 java.lang.UnsupportedOperationException: Cannot replace token 100284002935427428580945058996711341062 which does not exist!
2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:925) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61840 at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:389) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61840 at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61841 at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61910 Exception encountered during startup: Cannot replace token 100284002935427428580945058996711341062 which does not exist!
{code}
the status of the node is DN:
{code}
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
DN 2401:db00:2050:4196:face:0:13:0 809.83 GB 256 ? null ash5-04-pp
{code}
I add some logging and find something like this:
{code}
2016-05-25_20:58:33.44305 INFO 20:58:33 [main]: Gathering node replacement information for /2401:db00:2050:4196:face:0:13:0
2016-05-25_20:58:34.36966 INFO 20:58:34 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN
2016-05-25_20:58:41.12167 INFO 20:58:41 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN
2016-05-25_20:58:41.12248 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state STATUS
2016-05-25_20:58:41.12250 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 movename hibernate
2016-05-25_20:58:41.12252 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state LOAD
{code}
I find in the StorageService.onChange, we do not handle the "hibernate" VersionValue, does it cause the problem?
Is it safe to apply the patch to fix it?
was:
I got some errors when trying to replace a dead host.
{code}
2016-05-25_20:59:37.61838 ERROR 20:59:37 [main]: Exception encountered during startup
2016-05-25_20:59:37.61839 java.lang.UnsupportedOperationException: Cannot replace token 100284002935427428580945058996711341062 which does not exist!
2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:925) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61840 at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:389) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61840 at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61841 at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
2016-05-25_20:59:37.61910 Exception encountered during startup: Cannot replace token 100284002935427428580945058996711341062 which does not exist!
{code}
I add some logging and find something like this:
{code}
2016-05-25_20:58:33.44305 INFO 20:58:33 [main]: Gathering node replacement information for /2401:db00:2050:4196:face:0:13:0
2016-05-25_20:58:34.36966 INFO 20:58:34 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN
2016-05-25_20:58:41.12167 INFO 20:58:41 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN
2016-05-25_20:58:41.12248 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state STATUS
2016-05-25_20:58:41.12250 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 movename hibernate
2016-05-25_20:58:41.12252 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state LOAD
{code}
I find in the StorageService.onChange, we do not handle the "hibernate" VersionValue, does it cause the problem?
Is it safe to apply the patch to fix it?
> Can not replace a dead host
> ---------------------------
>
> Key: CASSANDRA-11892
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11892
> Project: Cassandra
> Issue Type: Bug
> Reporter: Dikang Gu
> Attachments: 0001-handle-hibernate-case.patch
>
>
> I got some errors when trying to replace a dead host.
> {code}
> 2016-05-25_20:59:37.61838 ERROR 20:59:37 [main]: Exception encountered during startup
> 2016-05-25_20:59:37.61839 java.lang.UnsupportedOperationException: Cannot replace token 100284002935427428580945058996711341062 which does not exist!
> 2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:925) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61839 at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61840 at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:389) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61840 at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61841 at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [apache-cassandra-2.1.14+git20160523.7442267.jar:2.1.14+git20160523.7442267]
> 2016-05-25_20:59:37.61910 Exception encountered during startup: Cannot replace token 100284002935427428580945058996711341062 which does not exist!
> {code}
> the status of the node is DN:
> {code}
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns Host ID Rack
> DN 2401:db00:2050:4196:face:0:13:0 809.83 GB 256 ? null ash5-04-pp
> {code}
> I add some logging and find something like this:
> {code}
> 2016-05-25_20:58:33.44305 INFO 20:58:33 [main]: Gathering node replacement information for /2401:db00:2050:4196:face:0:13:0
> 2016-05-25_20:58:34.36966 INFO 20:58:34 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN
> 2016-05-25_20:58:41.12167 INFO 20:58:41 [GossipStage:1]: InetAddress /2401:db00:2050:4196:face:0:13:0 is now DOWN
> 2016-05-25_20:58:41.12248 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state STATUS
> 2016-05-25_20:58:41.12250 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 movename hibernate
> 2016-05-25_20:58:41.12252 INFO 20:58:41 [GossipStage:1]: Node /2401:db00:2050:4196:face:0:13:0 state LOAD
> {code}
> I find in the StorageService.onChange, we do not handle the "hibernate" VersionValue, does it cause the problem?
> Is it safe to apply the patch to fix it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)