You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com> on 2014/03/12 21:57:58 UTC
Dead node seen as UP by replacement node
Hello,
I'm trying to replace a dead node using the procedure in [1], but the
replacement node initially sees the dead node as UP, and after a few
minutes the node is marked as DOWN again, failing the streaming/bootstrap
procedure of the replacement node. This dead node is always seen as DOWN by
the rest of the cluster.
Could this be a bug? I can easily reproduce it in our production
environment, but don't know if it's reproducible in a clean environment.
Version: 1.2.13
Here is the log from the replacement node (192.168.1.10 is the dead node):
INFO [GossipStage:1] 2014-03-12 20:25:41,089 Gossiper.java (line 843) Node
/192.168.1.10 is now part of the cluster
INFO [GossipStage:1] 2014-03-12 20:25:41,090 Gossiper.java (line 809)
InetAddress /192.168.1.10 is now UP
INFO [GossipTasks:1] 2014-03-12 20:34:54,238 Gossiper.java (line 823)
InetAddress /192.168.1.10 is now DOWN
ERROR [GossipTasks:1] 2014-03-12 20:34:54,240 AbstractStreamSession.java
(line 110) Stream failed because /192.168.1.10 died or was
restarted/removed (streams may still be active in background, but further
streams won't be started)
WARN [GossipTasks:1] 2014-03-12 20:34:54,240 RangeStreamer.java (line 246)
Streaming from /192.168.1.10 failed
ERROR [GossipTasks:1] 2014-03-12 20:34:54,240 AbstractStreamSession.java
(line 110) Stream failed because /192.168.1.10 died or was
restarted/removed (streams may still be active in background, but further
streams won't be started)
WARN [GossipTasks:1] 2014-03-12 20:34:54,241 RangeStreamer.java (line 246)
Streaming from /192.168.1.10 failed
[1]
http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node
Cheers,
Paulo
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
+55 83 9690-1314
Re: Dead node seen as UP by replacement node
Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
Hmm, we considered that option but if the old node is assassinated, his
range will be assigned to a neighbor that doesn't have the data, what will
cause empty reads. What we did to solve the problem was to do a safe
removal via "nodetool removenode <deadNodeId>", wait some hours for
neighbors to stream that node's data, and then bootstrap the replacement
node. However this procedure takes double the time, because data needs to
be streamed twice, which is not really optimal.
It would be really nice to know if this is expected behavior of if I should
fill a bug request.
On Fri, Mar 14, 2014 at 11:59 AM, Rahul Menon <ra...@apigee.com> wrote:
> Since the older node is not available i would ask you to assassinate the
> old node and then get the node new node to bootstrap.
>
>
> On Thu, Mar 13, 2014 at 10:56 PM, Paulo Ricardo Motta Gomes <
> paulo.motta@chaordicsystems.com> wrote:
>
>> Yes, exactly.
>>
>>
>> On Thu, Mar 13, 2014 at 1:27 PM, Rahul Menon <ra...@apigee.com> wrote:
>>
>>> And the token value as suggested is tokenvalueoddeadnode-1 ?
>>>
>>>
>>> On Thu, Mar 13, 2014 at 9:29 PM, Paulo Ricardo Motta Gomes <
>>> paulo.motta@chaordicsystems.com> wrote:
>>>
>>>> Nope, they have different IPs. I'm using the procedure described here
>>>> to replace a dead node:
>>>> http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node
>>>>
>>>> Dead node token: X (IP: Y)
>>>> Replacement node token: X-1 (IP: Z)
>>>>
>>>> So, as soon as the replacement node (Z) is started, it sees the dead
>>>> node (Y) as UP, and tries to stream data from it during the join process.
>>>> About 10 minutes later, the failure detector of Z detects Y as down, but
>>>> since it was trying to fetch data from him, it fails the join/bootstrap
>>>> process altogether.
>>>>
>>>>
>>>
>>
>>
>> --
>> *Paulo Motta*
>>
>> Chaordic | *Platform*
>> *www.chaordic.com.br <http://www.chaordic.com.br/>*
>> +55 48 3232.3200
>> +55 83 9690-1314
>>
>
>
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
+55 83 9690-1314
Re: Dead node seen as UP by replacement node
Posted by Rahul Menon <ra...@apigee.com>.
Since the older node is not available i would ask you to assassinate the
old node and then get the node new node to bootstrap.
On Thu, Mar 13, 2014 at 10:56 PM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:
> Yes, exactly.
>
>
> On Thu, Mar 13, 2014 at 1:27 PM, Rahul Menon <ra...@apigee.com> wrote:
>
>> And the token value as suggested is tokenvalueoddeadnode-1 ?
>>
>>
>> On Thu, Mar 13, 2014 at 9:29 PM, Paulo Ricardo Motta Gomes <
>> paulo.motta@chaordicsystems.com> wrote:
>>
>>> Nope, they have different IPs. I'm using the procedure described here to
>>> replace a dead node:
>>> http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node
>>>
>>> Dead node token: X (IP: Y)
>>> Replacement node token: X-1 (IP: Z)
>>>
>>> So, as soon as the replacement node (Z) is started, it sees the dead
>>> node (Y) as UP, and tries to stream data from it during the join process.
>>> About 10 minutes later, the failure detector of Z detects Y as down, but
>>> since it was trying to fetch data from him, it fails the join/bootstrap
>>> process altogether.
>>>
>>>
>>
>
>
> --
> *Paulo Motta*
>
> Chaordic | *Platform*
> *www.chaordic.com.br <http://www.chaordic.com.br/>*
> +55 48 3232.3200
> +55 83 9690-1314
>
Re: Dead node seen as UP by replacement node
Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
Yes, exactly.
On Thu, Mar 13, 2014 at 1:27 PM, Rahul Menon <ra...@apigee.com> wrote:
> And the token value as suggested is tokenvalueoddeadnode-1 ?
>
>
> On Thu, Mar 13, 2014 at 9:29 PM, Paulo Ricardo Motta Gomes <
> paulo.motta@chaordicsystems.com> wrote:
>
>> Nope, they have different IPs. I'm using the procedure described here to
>> replace a dead node:
>> http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node
>>
>> Dead node token: X (IP: Y)
>> Replacement node token: X-1 (IP: Z)
>>
>> So, as soon as the replacement node (Z) is started, it sees the dead node
>> (Y) as UP, and tries to stream data from it during the join process. About
>> 10 minutes later, the failure detector of Z detects Y as down, but since it
>> was trying to fetch data from him, it fails the join/bootstrap process
>> altogether.
>>
>>
>
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
+55 83 9690-1314
Re: Dead node seen as UP by replacement node
Posted by Rahul Menon <ra...@apigee.com>.
And the token value as suggested is tokenvalueoddeadnode-1 ?
On Thu, Mar 13, 2014 at 9:29 PM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:
> Nope, they have different IPs. I'm using the procedure described here to
> replace a dead node:
> http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node
>
> Dead node token: X (IP: Y)
> Replacement node token: X-1 (IP: Z)
>
> So, as soon as the replacement node (Z) is started, it sees the dead node
> (Y) as UP, and tries to stream data from it during the join process. About
> 10 minutes later, the failure detector of Z detects Y as down, but since it
> was trying to fetch data from him, it fails the join/bootstrap process
> altogether.
>
>
Re: Dead node seen as UP by replacement node
Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
Nope, they have different IPs. I'm using the procedure described here to
replace a dead node:
http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node
Dead node token: X (IP: Y)
Replacement node token: X-1 (IP: Z)
So, as soon as the replacement node (Z) is started, it sees the dead node
(Y) as UP, and tries to stream data from it during the join process. About
10 minutes later, the failure detector of Z detects Y as down, but since it
was trying to fetch data from him, it fails the join/bootstrap process
altogether.
Re: Dead node seen as UP by replacement node
Posted by Rahul Menon <ra...@apigee.com>.
Are you replacing the node with the same token and the same IP?
On Thu, Mar 13, 2014 at 4:36 AM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:
> Some further info:
>
> I'm not using Vnodes, so I'm using the 1.1 replace node trick of setting
> the initial_token in the cassandra.yaml file to the value of the dead
> node's token -1, and autobootstrap=true. However, according to the Apache
> wiki (
> https://wiki.apache.org/cassandra/Operations#For_versions_1.2.0_and_above),
> on 1.2 you should actually remove the dead node from the ring, before
> adding a replacement node.
>
> Does that mean the trick of setting the initial token to the value of the
> dead node's -1 (described in
> http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node) is
> not valid anymore in 1.2 without vnodes?
>
>
> On Wed, Mar 12, 2014 at 5:57 PM, Paulo Ricardo Motta Gomes <
> paulo.motta@chaordicsystems.com> wrote:
>
>> Hello,
>>
>> I'm trying to replace a dead node using the procedure in [1], but the
>> replacement node initially sees the dead node as UP, and after a few
>> minutes the node is marked as DOWN again, failing the streaming/bootstrap
>> procedure of the replacement node. This dead node is always seen as DOWN by
>> the rest of the cluster.
>>
>> Could this be a bug? I can easily reproduce it in our production
>> environment, but don't know if it's reproducible in a clean environment.
>>
>> Version: 1.2.13
>>
>> Here is the log from the replacement node (192.168.1.10 is the dead node):
>>
>> INFO [GossipStage:1] 2014-03-12 20:25:41,089 Gossiper.java (line 843)
>> Node /192.168.1.10 is now part of the cluster
>> INFO [GossipStage:1] 2014-03-12 20:25:41,090 Gossiper.java (line 809)
>> InetAddress /192.168.1.10 is now UP
>> INFO [GossipTasks:1] 2014-03-12 20:34:54,238 Gossiper.java (line 823)
>> InetAddress /192.168.1.10 is now DOWN
>> ERROR [GossipTasks:1] 2014-03-12 20:34:54,240 AbstractStreamSession.java
>> (line 110) Stream failed because /192.168.1.10 died or was
>> restarted/removed (streams may still be active in background, but further
>> streams won't be started)
>> WARN [GossipTasks:1] 2014-03-12 20:34:54,240 RangeStreamer.java (line
>> 246) Streaming from /192.168.1.10 failed
>> ERROR [GossipTasks:1] 2014-03-12 20:34:54,240 AbstractStreamSession.java
>> (line 110) Stream failed because /192.168.1.10 died or was
>> restarted/removed (streams may still be active in background, but further
>> streams won't be started)
>> WARN [GossipTasks:1] 2014-03-12 20:34:54,241 RangeStreamer.java (line
>> 246) Streaming from /192.168.1.10 failed
>>
>> [1]
>> http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node
>>
>>
>> Cheers,
>>
>> Paulo
>>
>> --
>> *Paulo Motta*
>>
>> Chaordic | *Platform*
>> *www.chaordic.com.br <http://www.chaordic.com.br/>*
>> +55 48 3232.3200
>> +55 83 9690-1314
>>
>
>
>
> --
> *Paulo Motta*
>
> Chaordic | *Platform*
> *www.chaordic.com.br <http://www.chaordic.com.br/>*
> +55 48 3232.3200
> +55 83 9690-1314
>
Re: Dead node seen as UP by replacement node
Posted by Paulo Ricardo Motta Gomes <pa...@chaordicsystems.com>.
Some further info:
I'm not using Vnodes, so I'm using the 1.1 replace node trick of setting
the initial_token in the cassandra.yaml file to the value of the dead
node's token -1, and autobootstrap=true. However, according to the Apache
wiki (
https://wiki.apache.org/cassandra/Operations#For_versions_1.2.0_and_above),
on 1.2 you should actually remove the dead node from the ring, before
adding a replacement node.
Does that mean the trick of setting the initial token to the value of the
dead node's -1 (described in
http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node) is
not valid anymore in 1.2 without vnodes?
On Wed, Mar 12, 2014 at 5:57 PM, Paulo Ricardo Motta Gomes <
paulo.motta@chaordicsystems.com> wrote:
> Hello,
>
> I'm trying to replace a dead node using the procedure in [1], but the
> replacement node initially sees the dead node as UP, and after a few
> minutes the node is marked as DOWN again, failing the streaming/bootstrap
> procedure of the replacement node. This dead node is always seen as DOWN by
> the rest of the cluster.
>
> Could this be a bug? I can easily reproduce it in our production
> environment, but don't know if it's reproducible in a clean environment.
>
> Version: 1.2.13
>
> Here is the log from the replacement node (192.168.1.10 is the dead node):
>
> INFO [GossipStage:1] 2014-03-12 20:25:41,089 Gossiper.java (line 843)
> Node /192.168.1.10 is now part of the cluster
> INFO [GossipStage:1] 2014-03-12 20:25:41,090 Gossiper.java (line 809)
> InetAddress /192.168.1.10 is now UP
> INFO [GossipTasks:1] 2014-03-12 20:34:54,238 Gossiper.java (line 823)
> InetAddress /192.168.1.10 is now DOWN
> ERROR [GossipTasks:1] 2014-03-12 20:34:54,240 AbstractStreamSession.java
> (line 110) Stream failed because /192.168.1.10 died or was
> restarted/removed (streams may still be active in background, but further
> streams won't be started)
> WARN [GossipTasks:1] 2014-03-12 20:34:54,240 RangeStreamer.java (line
> 246) Streaming from /192.168.1.10 failed
> ERROR [GossipTasks:1] 2014-03-12 20:34:54,240 AbstractStreamSession.java
> (line 110) Stream failed because /192.168.1.10 died or was
> restarted/removed (streams may still be active in background, but further
> streams won't be started)
> WARN [GossipTasks:1] 2014-03-12 20:34:54,241 RangeStreamer.java (line
> 246) Streaming from /192.168.1.10 failed
>
> [1]
> http://www.datastax.com/docs/1.1/cluster_management#replacing-a-dead-node
>
> Cheers,
>
> Paulo
>
> --
> *Paulo Motta*
>
> Chaordic | *Platform*
> *www.chaordic.com.br <http://www.chaordic.com.br/>*
> +55 48 3232.3200
> +55 83 9690-1314
>
--
*Paulo Motta*
Chaordic | *Platform*
*www.chaordic.com.br <http://www.chaordic.com.br/>*
+55 48 3232.3200
+55 83 9690-1314