You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Krish Donald <go...@gmail.com> on 2020/09/17 17:25:49 UTC
Node is UNREACHABLE after decommission
Hi,
We decommissioned a node from the cluster.
On decommissioned node it said in system.log that node has been
decommissioned .
But after couple of minutes only , on rest of the nodes the node is showing
UNREACHABLE when we issue nodetool describecluster .
nodetool status is not showing the node however nodetool describecluster is
showing UNREACHABLE.
I tried nodetool assassinate and now node is not showing in nodetool
describecluster , however that seems to be the last option.
Ideally it should leave the cluster immediately after decommission.
Once decommissioned is completed as per log then is there any issue in
issuing nodetool assasinate ?
Thanks
Re: Node is UNREACHABLE after decommission
Posted by onmstester onmstester <on...@zoho.com.INVALID>.
Another workaround that i used for UNREACHABLE nodes problem, is to restart the whole cluster and it would be fixed, but i don't know if it cause any problem or not
Sent using https://www.zoho.com/mail/
---- On Fri, 18 Sep 2020 01:19:35 +0430 Paulo Motta <pa...@gmail.com> wrote ----
Oh, if you're adding the same hosts to another cluster then the old cluster might try to communicate to the decommissioned nodes if you do that before the 3 day grace period. The cluster name not matching is a good protection, otherwise the two clusters will connect to each other and mayhem will ensue. You definitely don't want this to happen!
Unfortunately this delay of 3 days is hard-coded and non-configurable before 4.0 (see https://issues.apache.org/jira/browse/CASSANDRA-15596).
As long as all the old cluster nodes are UP and don't see the decommissioned node on "nodetool status", you can safely assassinate decommissioned nodes to prevent this.
The requirement that all nodes in the cluster are UP before safely assassinating a node is to prevent a down node from trying to connect to the decommissioned node after it recovers.
Em qui., 17 de set. de 2020 às 17:25, Krish Donald <ma...@gmail.com> escreveu:
Thanks Paulo,
We have to decommission multiple nodes from the cluster and move those nodes to other clusters.
So if we have to wait for 3 days for every node then it is going to take a lot of time.
If i am trying to add the decommissioned node to the other cluster it is giving me an error that cluster_name is not matching however cluster name is correct as per new cluster.
So until i issue assasinate , i am not able to move forward.
On Thu, Sep 17, 2020 at 1:13 PM Paulo Motta <ma...@gmail.com> wrote:
After decommissioning the node remains in gossip for a period of 3 days (if I recall correctly) and it will show up on describecluster during that period, so this is expected behavior. This allows other nodes that eventually were down when the node decommissioned to learn that this node left the cluster.
What assassinate does is remove the node from gossip, so that's why it no longer shows up on describecluster, but this shouldn't be necessary. You should check that the node successfully decommissioned if it doesn't show up on "nodetool status".
Em qui., 17 de set. de 2020 às 14:26, Krish Donald <ma...@gmail.com> escreveu:
We are on 3.11.5 opensource cassandra
On Thu, Sep 17, 2020 at 10:25 AM Krish Donald <ma...@gmail.com> wrote:
Hi,
We decommissioned a node from the cluster.
On decommissioned node it said in system.log that node has been decommissioned .
But after couple of minutes only , on rest of the nodes the node is showing UNREACHABLE when we issue nodetool describecluster .
nodetool status is not showing the node however nodetool describecluster is showing UNREACHABLE.
I tried nodetool assassinate and now node is not showing in nodetool describecluster , however that seems to be the last option.
Ideally it should leave the cluster immediately after decommission.
Once decommissioned is completed as per log then is there any issue in issuing nodetool assasinate ?
Thanks
Re: Node is UNREACHABLE after decommission
Posted by Paulo Motta <pa...@gmail.com>.
Oh, if you're adding the same hosts to another cluster then the old cluster
might try to communicate to the decommissioned nodes if you do that before
the 3 day grace period. The cluster name not matching is a good protection,
otherwise the two clusters will connect to each other and mayhem will
ensue. You definitely don't want this to happen!
Unfortunately this delay of 3 days is hard-coded and non-configurable
before 4.0 (see https://issues.apache.org/jira/browse/CASSANDRA-15596).
As long as all the old cluster nodes are UP and don't see the
decommissioned node on "nodetool status", you can safely assassinate
decommissioned nodes to prevent this.
The requirement that all nodes in the cluster are UP before safely
assassinating a node is to prevent a down node from trying to connect to
the decommissioned node after it recovers.
Em qui., 17 de set. de 2020 às 17:25, Krish Donald <go...@gmail.com>
escreveu:
> Thanks Paulo,
>
> We have to decommission multiple nodes from the cluster and move those
> nodes to other clusters.
> So if we have to wait for 3 days for every node then it is going to take a
> lot of time.
> If i am trying to add the decommissioned node to the other cluster it is
> giving me an error that cluster_name is not matching however cluster name
> is correct as per new cluster.
> So until i issue assasinate , i am not able to move forward.
>
> On Thu, Sep 17, 2020 at 1:13 PM Paulo Motta <pa...@gmail.com>
> wrote:
>
>> After decommissioning the node remains in gossip for a period of 3 days
>> (if I recall correctly) and it will show up on describecluster during that
>> period, so this is expected behavior. This allows other nodes that
>> eventually were down when the node decommissioned to learn that this node
>> left the cluster.
>>
>> What assassinate does is remove the node from gossip, so that's why it no
>> longer shows up on describecluster, but this shouldn't be necessary. You
>> should check that the node successfully decommissioned if it doesn't show
>> up on "nodetool status".
>>
>> Em qui., 17 de set. de 2020 às 14:26, Krish Donald <go...@gmail.com>
>> escreveu:
>>
>>> We are on 3.11.5 opensource cassandra
>>>
>>> On Thu, Sep 17, 2020 at 10:25 AM Krish Donald <go...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We decommissioned a node from the cluster.
>>>> On decommissioned node it said in system.log that node has been
>>>> decommissioned .
>>>> But after couple of minutes only , on rest of the nodes the node is
>>>> showing UNREACHABLE when we issue nodetool describecluster .
>>>>
>>>> nodetool status is not showing the node however nodetool
>>>> describecluster is showing UNREACHABLE.
>>>>
>>>> I tried nodetool assassinate and now node is not showing in nodetool
>>>> describecluster , however that seems to be the last option.
>>>>
>>>> Ideally it should leave the cluster immediately after decommission.
>>>> Once decommissioned is completed as per log then is there any issue in
>>>> issuing nodetool assasinate ?
>>>>
>>>> Thanks
>>>>
>>>>
Re: Node is UNREACHABLE after decommission
Posted by Krish Donald <go...@gmail.com>.
Thanks Paulo,
We have to decommission multiple nodes from the cluster and move those
nodes to other clusters.
So if we have to wait for 3 days for every node then it is going to take a
lot of time.
If i am trying to add the decommissioned node to the other cluster it is
giving me an error that cluster_name is not matching however cluster name
is correct as per new cluster.
So until i issue assasinate , i am not able to move forward.
On Thu, Sep 17, 2020 at 1:13 PM Paulo Motta <pa...@gmail.com>
wrote:
> After decommissioning the node remains in gossip for a period of 3 days
> (if I recall correctly) and it will show up on describecluster during that
> period, so this is expected behavior. This allows other nodes that
> eventually were down when the node decommissioned to learn that this node
> left the cluster.
>
> What assassinate does is remove the node from gossip, so that's why it no
> longer shows up on describecluster, but this shouldn't be necessary. You
> should check that the node successfully decommissioned if it doesn't show
> up on "nodetool status".
>
> Em qui., 17 de set. de 2020 às 14:26, Krish Donald <go...@gmail.com>
> escreveu:
>
>> We are on 3.11.5 opensource cassandra
>>
>> On Thu, Sep 17, 2020 at 10:25 AM Krish Donald <go...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> We decommissioned a node from the cluster.
>>> On decommissioned node it said in system.log that node has been
>>> decommissioned .
>>> But after couple of minutes only , on rest of the nodes the node is
>>> showing UNREACHABLE when we issue nodetool describecluster .
>>>
>>> nodetool status is not showing the node however nodetool describecluster
>>> is showing UNREACHABLE.
>>>
>>> I tried nodetool assassinate and now node is not showing in nodetool
>>> describecluster , however that seems to be the last option.
>>>
>>> Ideally it should leave the cluster immediately after decommission.
>>> Once decommissioned is completed as per log then is there any issue in
>>> issuing nodetool assasinate ?
>>>
>>> Thanks
>>>
>>>
Re: Node is UNREACHABLE after decommission
Posted by Paulo Motta <pa...@gmail.com>.
After decommissioning the node remains in gossip for a period of 3 days (if
I recall correctly) and it will show up on describecluster during that
period, so this is expected behavior. This allows other nodes that
eventually were down when the node decommissioned to learn that this node
left the cluster.
What assassinate does is remove the node from gossip, so that's why it no
longer shows up on describecluster, but this shouldn't be necessary. You
should check that the node successfully decommissioned if it doesn't show
up on "nodetool status".
Em qui., 17 de set. de 2020 às 14:26, Krish Donald <go...@gmail.com>
escreveu:
> We are on 3.11.5 opensource cassandra
>
> On Thu, Sep 17, 2020 at 10:25 AM Krish Donald <go...@gmail.com>
> wrote:
>
>> Hi,
>>
>> We decommissioned a node from the cluster.
>> On decommissioned node it said in system.log that node has been
>> decommissioned .
>> But after couple of minutes only , on rest of the nodes the node is
>> showing UNREACHABLE when we issue nodetool describecluster .
>>
>> nodetool status is not showing the node however nodetool describecluster
>> is showing UNREACHABLE.
>>
>> I tried nodetool assassinate and now node is not showing in nodetool
>> describecluster , however that seems to be the last option.
>>
>> Ideally it should leave the cluster immediately after decommission.
>> Once decommissioned is completed as per log then is there any issue in
>> issuing nodetool assasinate ?
>>
>> Thanks
>>
>>
Re: Node is UNREACHABLE after decommission
Posted by Krish Donald <go...@gmail.com>.
We are on 3.11.5 opensource cassandra
On Thu, Sep 17, 2020 at 10:25 AM Krish Donald <go...@gmail.com> wrote:
> Hi,
>
> We decommissioned a node from the cluster.
> On decommissioned node it said in system.log that node has been
> decommissioned .
> But after couple of minutes only , on rest of the nodes the node is
> showing UNREACHABLE when we issue nodetool describecluster .
>
> nodetool status is not showing the node however nodetool describecluster
> is showing UNREACHABLE.
>
> I tried nodetool assassinate and now node is not showing in nodetool
> describecluster , however that seems to be the last option.
>
> Ideally it should leave the cluster immediately after decommission.
> Once decommissioned is completed as per log then is there any issue in
> issuing nodetool assasinate ?
>
> Thanks
>
>