You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by rajasekhar kommineni <ra...@gmail.com> on 2018/07/12 17:45:57 UTC

Best approach for node decommission

Hi All,

Can anybody let me know best approach for decommissiong a node in the cluster. My cluster is using vnodes, is there any way to verify all the data of the decommissioning node has been moved to remaining nodes, before completely shutting down the server.

I followed below procedure :

1) nodetool flush
2) nodetool repair
3) nodetool decommission

The aggregate of Load before node3 decommission is 1411.47 and after is 1380.15. Can I ignore the size difference and treat all the data of node3 has been moved to other nodes.

I am looking for good data validation process with out depending on Application team for verification.

Total load : 1411.47

– Address Load Tokens Owns Host ID Rack
UN node1 220.48 MiB 256 ? ff09b08b-29c1-4365-a3b7-1eea51f7d575 rack1
UN node2 216.53 MiB 256 ? 4b565a31-4c77-418f-a47f-5e0eb2ec5624 rack1
UN node3 64.52  MiB 256 ? 12b29812-cc60-456c-95a9-0e339c249bc8 rack1
UN node4 195.84 MiB 256 ? 0424a882-de4f-4e6a-b642-6ce9f4621e04 rack1
UN node5 179.07 MiB 256 ? 2f291a2e-b10d-4364-8192-13e107a9c322 rack1
UN node6 213.75 MiB 256 ? cf10166b-cfae-44fd-8bca-f55a4f9ef491 rack1
UN node7 158.54 MiB 256 ? ef8454c7-3005-487a-a3d4-e0065edfd99f rack1
UN node8 162.74 MiB 256 ? 7d786e46-1c11-485c-a943-bbcca6729ae1 rack1

Total Load : 1380.15

– Address Load Tokens Owns Host ID Rack
UN node1 229.04 MiB 256 ? ff09b08b-29c1-4365-a3b7-1eea51f7d575 rack1
UN node2 225.52 MiB 256 ? 4b565a31-4c77-418f-a47f-5e0eb2ec5624 rack1
UN node4 195.84 MiB 256 ? 0424a882-de4f-4e6a-b642-6ce9f4621e04 rack1
UN node5 179.07 MiB 256 ? 2f291a2e-b10d-4364-8192-13e107a9c322 rack1
UN node6 229.4  MiB 256 ? cf10166b-cfae-44fd-8bca-f55a4f9ef491 rack1
UN node7 158.54 MiB 256 ? ef8454c7-3005-487a-a3d4-e0065edfd99f rack1
UN node8 162.74 MiB 256 ? 7d786e46-1c11-485c-a943-bbcca6729ae1 rack1

Thanks,

Re: Best approach for node decommission

Posted by Soumya Jena <so...@gmail.com>.
If it helps you can use 'nodetool netstats' to monitor . If you do not see
any streams stuck there , then you should be good . Also if you are
concerned , you can run "nodetool repair" after the decommission just to
make sure .

I think what you are doing is okay . If the node is alive then  nodetool
decommission is best approach  . If the node is dead then you have to use
nodetool removenode <Host ID>

On Thu, Jul 12, 2018 at 10:45 AM, rajasekhar kommineni <ra...@gmail.com>
wrote:

> Hi All,
>
> Can anybody let me know best approach for decommissiong a node in the
> cluster. My cluster is using vnodes, is there any way to verify all the
> data of the decommissioning node has been moved to remaining nodes, before
> completely shutting down the server.
>
> I followed below procedure :
>
> 1) nodetool flush
> 2) nodetool repair
> 3) nodetool decommission
>
> The aggregate of Load before node3 decommission is 1411.47 and after is
> 1380.15. Can I ignore the size difference and treat all the data of node3
> has been moved to other nodes.
>
> I am looking for good data validation process with out depending on
> Application team for verification.
>
> *Total load : 1411.47*
>
> – Address Load Tokens Owns Host ID Rack
> UN node1 220.48 MiB 256 ? ff09b08b-29c1-4365-a3b7-1eea51f7d575 rack1
> UN node2 216.53 MiB 256 ? 4b565a31-4c77-418f-a47f-5e0eb2ec5624 rack1
> UN node3 64.52  MiB 256 ? 12b29812-cc60-456c-95a9-0e339c249bc8 rack1
> UN node4 195.84 MiB 256 ? 0424a882-de4f-4e6a-b642-6ce9f4621e04 rack1
> UN node5 179.07 MiB 256 ? 2f291a2e-b10d-4364-8192-13e107a9c322 rack1
> UN node6 213.75 MiB 256 ? cf10166b-cfae-44fd-8bca-f55a4f9ef491 rack1
> UN node7 158.54 MiB 256 ? ef8454c7-3005-487a-a3d4-e0065edfd99f rack1
> UN node8 162.74 MiB 256 ? 7d786e46-1c11-485c-a943-bbcca6729ae1 rack1
>
> *Total Load : 1380.15*
>
> – Address Load Tokens Owns Host ID Rack
> UN node1 229.04 MiB 256 ? ff09b08b-29c1-4365-a3b7-1eea51f7d575 rack1
> UN node2 225.52 MiB 256 ? 4b565a31-4c77-418f-a47f-5e0eb2ec5624 rack1
> UN node4 195.84 MiB 256 ? 0424a882-de4f-4e6a-b642-6ce9f4621e04 rack1
> UN node5 179.07 MiB 256 ? 2f291a2e-b10d-4364-8192-13e107a9c322 rack1
> UN node6 229.4  MiB 256 ? cf10166b-cfae-44fd-8bca-f55a4f9ef491 rack1
> UN node7 158.54 MiB 256 ? ef8454c7-3005-487a-a3d4-e0065edfd99f rack1
> UN node8 162.74 MiB 256 ? 7d786e46-1c11-485c-a943-bbcca6729ae1 rack1
>
> Thanks,
>
>