You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mark Furlong <mf...@ancestry.com> on 2017/05/16 16:01:35 UTC
Decommissioned node cluster shows as down
I have a node I decommissioned on a large ring using 2.1.12. The node completed the decommission process and is no longer communicating with the rest of the cluster. However when I run a nodetool status on any node in the cluster it shows the node as ‘DN’. Why is this and should I just run a removenode now?
Thanks,
Mark Furlong
Sr. Database Administrator
mfurlong@ancestry.com<ma...@ancestry.com>
M: 801-859-7427
O: 801-705-7115
1300 W Traverse Pkwy
Lehi, UT 84043
[http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]
Re: Decommissioned node cluster shows as down
Posted by Gedeon Kamga <gk...@gmail.com>.
If the nodetool status shows DN, then you will need to tell the cluster to
remove the memory of that node by running removenode. If that doesn't work,
you may run assassinate - the ultimate command to shutdown the node.
Thanks,
Gedeon
On Tue, May 16, 2017 at 12:01 PM, Mark Furlong <mf...@ancestry.com>
wrote:
> I have a node I decommissioned on a large ring using 2.1.12. The node
> completed the decommission process and is no longer communicating with the
> rest of the cluster. However when I run a nodetool status on any node in
> the cluster it shows the node as ‘DN’. Why is this and should I just run a
> removenode now?
>
>
>
> *Thanks,*
>
> *Mark Furlong*
>
> Sr. Database Administrator
>
> *mfurlong@ancestry.com <mf...@ancestry.com>*
> M: 801-859-7427 <(801)%20859-7427>
>
> O: 801-705-7115 <(801)%20705-7115>
>
> 1300 W Traverse Pkwy
>
> Lehi, UT 84043
>
>
>
>
>
> [image: http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]
>
>
>
>
>
Re: RE: Decommissioned node cluster shows as down
Posted by Jeff Jirsa <jj...@apache.org>.
On 2017-05-16 09:28 (-0700), Mark Furlong <mf...@ancestry.com> wrote:
> I thought the same that the decommission would complete the removal of a node. I have heard something said about a 72 hour window, Iâm not sure if that pertains to this version.
>
We keep a record of it in gossip for 72 hours (in a special status to indicate that the node left the ring), just in case we had a host that was offline during the decommission come back to life and still have a record of that now-removed host in its saved system tables. It'll be in gossip, but it shouldn't be in 'nodetool ring' or 'nodetool status' output.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
RE: Decommissioned node cluster shows as down
Posted by Mark Furlong <mf...@ancestry.com>.
I thought the same that the decommission would complete the removal of a node. I have heard something said about a 72 hour window, I’m not sure if that pertains to this version.
Thanks
Mark
801-705-7115 office
From: Hannu Kröger [mailto:hkroger@gmail.com]
Sent: Tuesday, May 16, 2017 10:09 AM
To: suraj pasuparthy <su...@gmail.com>
Cc: Mark Furlong <mf...@ancestry.com>; user@cassandra.apache.org
Subject: Re: Decommissioned node cluster shows as down
That’s weird. I thought decommission would ultimately remove the node from the cluster because the token(s) should be removed from the ring and data should be streamed to new owners. “DN” is IMHO not a state where the node should end up in.
Hannu
On 16 May 2017, at 19:05, suraj pasuparthy <su...@gmail.com>> wrote:
Yes, you have to run a nodetool removenode to decomission completely.. this will also allow another node with the same ip different HashId to join the cluster..
Thanks
-suraj
On Tue, May 16, 2017 at 9:01 AM Mark Furlong <mf...@ancestry.com>> wrote:
I have a node I decommissioned on a large ring using 2.1.12. The node completed the decommission process and is no longer communicating with the rest of the cluster. However when I run a nodetool status on any node in the cluster it shows
the node as ‘DN’. Why is this and should I just run a removenode now?
Thanks,
Mark Furlong
Sr. Database Administrator
mfurlong@ancestry.com<ma...@ancestry.com>
M: 801-859-7427
O: 801-705-7115
1300 W Traverse Pkwy
Lehi, UT 84043
<image003.png>
Re: Decommissioned node cluster shows as down
Posted by Jeff Jirsa <jj...@apache.org>.
On 2017-05-16 09:08 (-0700), Hannu Kröger <hk...@gmail.com> wrote:
> Thatâs weird. I thought decommission would ultimately remove the node from the cluster because the token(s) should be removed from the ring and data should be streamed to new owners. âDNâ is IMHO not a state where the node should end up in.
>
Decommission should remove the node from the cluster, it should not be in "DN" if decommission finished successfully.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Decommissioned node cluster shows as down
Posted by Hannu Kröger <hk...@gmail.com>.
That’s weird. I thought decommission would ultimately remove the node from the cluster because the token(s) should be removed from the ring and data should be streamed to new owners. “DN” is IMHO not a state where the node should end up in.
Hannu
> On 16 May 2017, at 19:05, suraj pasuparthy <su...@gmail.com> wrote:
>
> Yes, you have to run a nodetool removenode to decomission completely.. this will also allow another node with the same ip different HashId to join the cluster..
>
> Thanks
> -suraj
> On Tue, May 16, 2017 at 9:01 AM Mark Furlong <mfurlong@ancestry.com <ma...@ancestry.com>> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I have a node I decommissioned on a large ring using 2.1.12. The node completed the decommission process and is no longer communicating with the rest of the cluster. However when I run a nodetool status on any node in the cluster it shows
>
> the node as ‘DN’. Why is this and should I just run a removenode now?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Thanks,
>
>
>
> Mark Furlong
>
>
>
>
>
>
>
> Sr. Database Administrator
>
>
>
>
>
>
>
> mfurlong@ancestry.com <ma...@ancestry.com>
>
>
> M: 801-859-7427
>
>
>
> O: 801-705-7115
>
>
>
> 1300 W Traverse Pkwy
>
>
>
> Lehi, UT 84043
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> <image003.png>
Re: Decommissioned node cluster shows as down
Posted by suraj pasuparthy <su...@gmail.com>.
Yes, you have to run a nodetool removenode to decomission completely.. this
will also allow another node with the same ip different HashId to join the
cluster..
Thanks
-suraj
On Tue, May 16, 2017 at 9:01 AM Mark Furlong <mf...@ancestry.com> wrote:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I have a node I decommissioned on a large ring using 2.1.12. The node
> completed the decommission process and is no longer communicating with the
> rest of the cluster. However when I run a nodetool status on any node in
> the cluster it shows
>
> the node as ‘DN’. Why is this and should I just run a removenode now?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Thanks,*
>
>
> *Mark Furlong*
>
>
>
>
>
>
>
> Sr. Database Administrator
>
>
>
>
>
>
>
> *mfurlong@ancestry.com <mf...@ancestry.com>*
>
>
> M: 801-859-7427
>
>
> O: 801-705-7115
>
>
> 1300 W Traverse Pkwy
>
>
> Lehi, UT 84043
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> [image: http://c.mfcreative.com/mars/email/shared-icon/sig-logo.gif]
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>