You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dikang Gu <di...@gmail.com> on 2016/11/18 19:02:38 UTC
Node replacement failed in 2.2
Hi, I encountered couple times that I could not replace a down node due to
error:
2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException)
encountered during startup: Could not find tokens for
/2401:db00:2130:4091:face:0:13:0 to replace
2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered
during startup
2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could not find tokens
for /2401:db00:2130:4091:face:0:13:0 to replace
2016-11-17_19:33:58.70491 at
org.apache.cassandra.service.StorageService.prepareReplacementInfo(StorageService.java:525)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-11-17_19:33:58.70492 at
org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:760)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-11-17_19:33:58.70492 at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:693)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-11-17_19:33:58.70492 at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:585)
~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-11-17_19:33:58.70492 at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300)
[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-11-17_19:33:58.70493 at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516)
[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-11-17_19:33:58.70493 at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625)
[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
2016-11-17_19:33:58.70649 INFO 19:33:58 [StorageServiceShutdownHook]:
Announcing shutdown
2016-11-17_19:34:00.70967 INFO 19:34:00 [StorageServiceShutdownHook]:
Waiting for messaging service to quiesce
2016-11-17_19:34:00.71066 INFO 19:34:00
[ACCEPT-/2401:db00:2130:4091:face:0:13:0]: MessagingService has terminated
the accept() thread
Did not find a relevant ticket for this, is anyone aware of this?
Thanks!
--
Dikang
Re: Node replacement failed in 2.2
Posted by Dikang Gu <di...@gmail.com>.
Hmm, I don't think we use join_ring=false or write_survey=true for that
node. I already remove_node to take the bad node out of ring, will try to
have more debug logs next time.
Thanks.
On Sun, Nov 20, 2016 at 2:31 PM, Paulo Motta <pa...@gmail.com>
wrote:
> Is there any chance the replaced node recently resumed bootstrap, joined
> with join_ring=false or write_survey=true? If so, perhaps this could be
> related to CASSANDRA-12935.
>
> Otherwise gossip tokens being empty is definitely unexpected behavior and
> you should probably file another ticket with more details/context (such as
> gossip debug logs of replacement and other nodes, and if the replacement
> node had the same or different ip as the original node since they are
> slightly different code paths after #8523).
>
> 2016-11-18 19:07 GMT-02:00 Dikang Gu <di...@gmail.com>:
>
>> Paulo, the tokens field for 2401:db00:2130:4091:face:0:13:0 shows
>> "TOKENS: not present", on all live nodes. It means tokens are missing,
>> right? What would cause this?
>>
>> Thanks.
>> Dikang.
>>
>> On Fri, Nov 18, 2016 at 11:15 AM, Paulo Motta <pa...@gmail.com>
>> wrote:
>>
>>> What does nodetool gossipinfo shows for endpoint /2401:db00:2130:4091:
>>> face:0:13:0 ? Does it contain the TOKENS attribute? If it's missing, is
>>> it only missing on this node or other nodes as well?
>>>
>>> 2016-11-18 17:02 GMT-02:00 Dikang Gu <di...@gmail.com>:
>>>
>>>> Hi, I encountered couple times that I could not replace a down node due
>>>> to error:
>>>>
>>>> 2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException)
>>>> encountered during startup: Could not find tokens for
>>>> /2401:db00:2130:4091:face:0:13:0 to replace
>>>> 2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered
>>>> during startup
>>>> 2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could not find
>>>> tokens for /2401:db00:2130:4091:face:0:13:0 to replace
>>>> 2016-11-17_19:33:58.70491 at org.apache.cassandra.service.S
>>>> torageService.prepareReplacementInfo(StorageService.java:525)
>>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>>> 160315.c29948b]
>>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S
>>>> torageService.prepareToJoin(StorageService.java:760)
>>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>>> 160315.c29948b]
>>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S
>>>> torageService.initServer(StorageService.java:693)
>>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>>> 160315.c29948b]
>>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S
>>>> torageService.initServer(StorageService.java:585)
>>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>>> 160315.c29948b]
>>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.C
>>>> assandraDaemon.setup(CassandraDaemon.java:300)
>>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>>>> 60315.c29948b]
>>>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C
>>>> assandraDaemon.activate(CassandraDaemon.java:516)
>>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>>>> 60315.c29948b]
>>>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C
>>>> assandraDaemon.main(CassandraDaemon.java:625)
>>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>>>> 60315.c29948b]
>>>> 2016-11-17_19:33:58.70649 INFO 19:33:58 [StorageServiceShutdownHook]:
>>>> Announcing shutdown
>>>> 2016-11-17_19:34:00.70967 INFO 19:34:00 [StorageServiceShutdownHook]:
>>>> Waiting for messaging service to quiesce
>>>> 2016-11-17_19:34:00.71066 INFO 19:34:00 [ACCEPT-/2401:db00:2130:4091:face:0:13:0]:
>>>> MessagingService has terminated the accept() thread
>>>>
>>>> Did not find a relevant ticket for this, is anyone aware of this?
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> Dikang
>>>>
>>>>
>>>
>>
>>
>> --
>> Dikang
>>
>>
>
--
Dikang
Re: Node replacement failed in 2.2
Posted by Paulo Motta <pa...@gmail.com>.
Is there any chance the replaced node recently resumed bootstrap, joined
with join_ring=false or write_survey=true? If so, perhaps this could be
related to CASSANDRA-12935.
Otherwise gossip tokens being empty is definitely unexpected behavior and
you should probably file another ticket with more details/context (such as
gossip debug logs of replacement and other nodes, and if the replacement
node had the same or different ip as the original node since they are
slightly different code paths after #8523).
2016-11-18 19:07 GMT-02:00 Dikang Gu <di...@gmail.com>:
> Paulo, the tokens field for 2401:db00:2130:4091:face:0:13:0 shows
> "TOKENS: not present", on all live nodes. It means tokens are missing,
> right? What would cause this?
>
> Thanks.
> Dikang.
>
> On Fri, Nov 18, 2016 at 11:15 AM, Paulo Motta <pa...@gmail.com>
> wrote:
>
>> What does nodetool gossipinfo shows for endpoint /2401:db00:2130:4091:
>> face:0:13:0 ? Does it contain the TOKENS attribute? If it's missing, is
>> it only missing on this node or other nodes as well?
>>
>> 2016-11-18 17:02 GMT-02:00 Dikang Gu <di...@gmail.com>:
>>
>>> Hi, I encountered couple times that I could not replace a down node due
>>> to error:
>>>
>>> 2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException)
>>> encountered during startup: Could not find tokens for
>>> /2401:db00:2130:4091:face:0:13:0 to replace
>>> 2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered
>>> during startup
>>> 2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could not find
>>> tokens for /2401:db00:2130:4091:face:0:13:0 to replace
>>> 2016-11-17_19:33:58.70491 at org.apache.cassandra.service.S
>>> torageService.prepareReplacementInfo(StorageService.java:525)
>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>> 160315.c29948b]
>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S
>>> torageService.prepareToJoin(StorageService.java:760)
>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>> 160315.c29948b]
>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S
>>> torageService.initServer(StorageService.java:693)
>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>> 160315.c29948b]
>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S
>>> torageService.initServer(StorageService.java:585)
>>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>>> 160315.c29948b]
>>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.C
>>> assandraDaemon.setup(CassandraDaemon.java:300)
>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>>> 60315.c29948b]
>>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C
>>> assandraDaemon.activate(CassandraDaemon.java:516)
>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>>> 60315.c29948b]
>>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C
>>> assandraDaemon.main(CassandraDaemon.java:625)
>>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>>> 60315.c29948b]
>>> 2016-11-17_19:33:58.70649 INFO 19:33:58 [StorageServiceShutdownHook]:
>>> Announcing shutdown
>>> 2016-11-17_19:34:00.70967 INFO 19:34:00 [StorageServiceShutdownHook]:
>>> Waiting for messaging service to quiesce
>>> 2016-11-17_19:34:00.71066 INFO 19:34:00 [ACCEPT-/2401:db00:2130:4091:face:0:13:0]:
>>> MessagingService has terminated the accept() thread
>>>
>>> Did not find a relevant ticket for this, is anyone aware of this?
>>>
>>> Thanks!
>>>
>>> --
>>> Dikang
>>>
>>>
>>
>
>
> --
> Dikang
>
>
Re: Node replacement failed in 2.2
Posted by Dikang Gu <di...@gmail.com>.
Paulo, the tokens field for 2401:db00:2130:4091:face:0:13:0 shows "TOKENS:
not present", on all live nodes. It means tokens are missing, right? What
would cause this?
Thanks.
Dikang.
On Fri, Nov 18, 2016 at 11:15 AM, Paulo Motta <pa...@gmail.com>
wrote:
> What does nodetool gossipinfo shows for endpoint /2401:db00:2130:4091:
> face:0:13:0 ? Does it contain the TOKENS attribute? If it's missing, is
> it only missing on this node or other nodes as well?
>
> 2016-11-18 17:02 GMT-02:00 Dikang Gu <di...@gmail.com>:
>
>> Hi, I encountered couple times that I could not replace a down node due
>> to error:
>>
>> 2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException)
>> encountered during startup: Could not find tokens for
>> /2401:db00:2130:4091:face:0:13:0 to replace
>> 2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered
>> during startup
>> 2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could not find
>> tokens for /2401:db00:2130:4091:face:0:13:0 to replace
>> 2016-11-17_19:33:58.70491 at org.apache.cassandra.service.S
>> torageService.prepareReplacementInfo(StorageService.java:525)
>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>> 160315.c29948b]
>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S
>> torageService.prepareToJoin(StorageService.java:760)
>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>> 160315.c29948b]
>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S
>> torageService.initServer(StorageService.java:693)
>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>> 160315.c29948b]
>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.S
>> torageService.initServer(StorageService.java:585)
>> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20
>> 160315.c29948b]
>> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.C
>> assandraDaemon.setup(CassandraDaemon.java:300)
>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>> 60315.c29948b]
>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C
>> assandraDaemon.activate(CassandraDaemon.java:516)
>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>> 60315.c29948b]
>> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.C
>> assandraDaemon.main(CassandraDaemon.java:625)
>> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git201
>> 60315.c29948b]
>> 2016-11-17_19:33:58.70649 INFO 19:33:58 [StorageServiceShutdownHook]:
>> Announcing shutdown
>> 2016-11-17_19:34:00.70967 INFO 19:34:00 [StorageServiceShutdownHook]:
>> Waiting for messaging service to quiesce
>> 2016-11-17_19:34:00.71066 INFO 19:34:00 [ACCEPT-/2401:db00:2130:4091:face:0:13:0]:
>> MessagingService has terminated the accept() thread
>>
>> Did not find a relevant ticket for this, is anyone aware of this?
>>
>> Thanks!
>>
>> --
>> Dikang
>>
>>
>
--
Dikang
Re: Node replacement failed in 2.2
Posted by Paulo Motta <pa...@gmail.com>.
What does nodetool gossipinfo shows for endpoint
/2401:db00:2130:4091:face:0:13:0 ? Does it contain the TOKENS attribute? If
it's missing, is it only missing on this node or other nodes as well?
2016-11-18 17:02 GMT-02:00 Dikang Gu <di...@gmail.com>:
> Hi, I encountered couple times that I could not replace a down node due to
> error:
>
> 2016-11-17_19:33:58.70075 Exception (java.lang.RuntimeException)
> encountered during startup: Could not find tokens for
> /2401:db00:2130:4091:face:0:13:0 to replace
> 2016-11-17_19:33:58.70489 ERROR 19:33:58 [main]: Exception encountered
> during startup
> 2016-11-17_19:33:58.70491 java.lang.RuntimeException: Could not find
> tokens for /2401:db00:2130:4091:face:0:13:0 to replace
> 2016-11-17_19:33:58.70491 at org.apache.cassandra.service.
> StorageService.prepareReplacementInfo(StorageService.java:525)
> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+
> git20160315.c29948b]
> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.
> StorageService.prepareToJoin(StorageService.java:760)
> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+
> git20160315.c29948b]
> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.
> StorageService.initServer(StorageService.java:693)
> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+
> git20160315.c29948b]
> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.
> StorageService.initServer(StorageService.java:585)
> ~[apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+
> git20160315.c29948b]
> 2016-11-17_19:33:58.70492 at org.apache.cassandra.service.
> CassandraDaemon.setup(CassandraDaemon.java:300) [apache-cassandra-2.2.5+
> git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.
> CassandraDaemon.activate(CassandraDaemon.java:516)
> [apache-cassandra-2.2.5+git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-11-17_19:33:58.70493 at org.apache.cassandra.service.
> CassandraDaemon.main(CassandraDaemon.java:625) [apache-cassandra-2.2.5+
> git20160315.c29948b.jar:2.2.5+git20160315.c29948b]
> 2016-11-17_19:33:58.70649 INFO 19:33:58 [StorageServiceShutdownHook]:
> Announcing shutdown
> 2016-11-17_19:34:00.70967 INFO 19:34:00 [StorageServiceShutdownHook]:
> Waiting for messaging service to quiesce
> 2016-11-17_19:34:00.71066 INFO 19:34:00 [ACCEPT-/2401:db00:2130:4091:face:0:13:0]:
> MessagingService has terminated the accept() thread
>
> Did not find a relevant ticket for this, is anyone aware of this?
>
> Thanks!
>
> --
> Dikang
>
>