You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Baskar Duraikannu <ba...@gmail.com> on 2011/04/12 18:53:31 UTC
Help on decommission
I have setup a 4 node cluster for testing. When I setup the cluster, I have setup initial tokens in such a way that each gets 25% of load and then started the node with autobootstrap=false.
After all nodes are up, I loaded data using the stress test tool with replication factor of 3. As per of my testing, I am trying to remove one of the node using "nodetool decomission" but the node seems to be stuck in "leaving" status.
How do I check whether it is doing any work at all? Please help
[root@localhost bin]# ./nodetool -h 10.140.22.25 ring
Address Status State Load Owns Token
127605887595351923798765477786913079296
10.140.22.66 Up Leaving 119.41 MB 25.00% 0
10.140.22.42 Up Normal 116.23 MB 25.00% 42535295865117307932921825928971026432
10.140.22.28 Up Normal 119.93 MB 25.00% 85070591730234615865843651857942052864
10.140.22.25 Up Normal 116.21 MB 25.00% 127605887595351923798765477786913079296
[root@localhost bin]# ./nodetool -h 10.140.22.66 netstats
Mode: Leaving: streaming data to other nodes
Streaming to: /10.140.22.42
/var/lib/cassandra/data/Keyspace1/Standard1-f-1-Data.db/(0,120929157)
progress=120929157/120929157 - 100%
/var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291)
progress=0/3361291 - 0%
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 0 17
Responses n/a 0 108109
[root@usnynyc1cass02 bin]# ./nodetool -h 10.140.22.42 netstats
Mode: Normal
Not sending any streams.
Streaming from: /10.140.22.66
Keyspace1: /var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291)
progress=0/3361291 - 0%
Pool Name Active Pending Completed
Commands n/a 0 11
Responses n/a 0 107879
Regards,
Baskar
Re: Help on decommission
Posted by Baskar Duraikannu <ba...@gmail.com>.
No. I stopped the stress test before issuing decommission command. So, it was not under ANY load.
I waited for over an hour and nothing changed.
Then , I turned on DEBUG in the log4j-server.properties and then restarted the Cassandra process .
As soon as I restarted, the decommissioned node left the cluster and everything was back to normal.
Have you seen this behaviour before?
From: Jonathan Colby
Sent: Tuesday, April 12, 2011 3:15 PM
To: user@cassandra.apache.org
Subject: Re: Help on decommission
how long as it been in Leaving status? Is the cluster under stress test load while you are doing the decommission?
On Apr 12, 2011, at 6:53 PM, Baskar Duraikannu wrote:
I have setup a 4 node cluster for testing. When I setup the cluster, I have setup initial tokens in such a way that each gets 25% of load and then started the node with autobootstrap=false.
After all nodes are up, I loaded data using the stress test tool with replication factor of 3. As per of my testing, I am trying to remove one of the node using "nodetool decomission" but the node seems to be stuck in "leaving" status.
How do I check whether it is doing any work at all? Please help
[root@localhost bin]# ./nodetool -h 10.140.22.25 ring
Address Status State Load Owns Token
127605887595351923798765477786913079296
10.140.22.66 Up Leaving 119.41 MB 25.00% 0
10.140.22.42 Up Normal 116.23 MB 25.00% 42535295865117307932921825928971026432
10.140.22.28 Up Normal 119.93 MB 25.00% 85070591730234615865843651857942052864
10.140.22.25 Up Normal 116.21 MB 25.00% 127605887595351923798765477786913079296
[root@localhost bin]# ./nodetool -h 10.140.22.66 netstats
Mode: Leaving: streaming data to other nodes
Streaming to: /10.140.22.42
/var/lib/cassandra/data/Keyspace1/Standard1-f-1-Data.db/(0,120929157)
progress=120929157/120929157 - 100%
/var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291)
progress=0/3361291 - 0%
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 0 17
Responses n/a 0 108109
[root@usnynyc1cass02 bin]# ./nodetool -h 10.140.22.42 netstats
Mode: Normal
Not sending any streams.
Streaming from: /10.140.22.66
Keyspace1: /var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291)
progress=0/3361291 - 0%
Pool Name Active Pending Completed
Commands n/a 0 11
Responses n/a 0 107879
Regards,
Baskar
Re: Help on decommission
Posted by Jonathan Colby <jo...@gmail.com>.
how long as it been in Leaving status? Is the cluster under stress test load while you are doing the decommission?
On Apr 12, 2011, at 6:53 PM, Baskar Duraikannu wrote:
> I have setup a 4 node cluster for testing. When I setup the cluster, I have setup initial tokens in such a way that each gets 25% of load and then started the node with autobootstrap=false.
>
>
> After all nodes are up, I loaded data using the stress test tool with replication factor of 3. As per of my testing, I am trying to remove one of the node using "nodetool decomission" but the node seems to be stuck in "leaving" status.
>
> How do I check whether it is doing any work at all? Please help
>
>
> [root@localhost bin]# ./nodetool -h 10.140.22.25 ring
> Address Status State Load Owns Token
> 127605887595351923798765477786913079296
> 10.140.22.66 Up Leaving 119.41 MB 25.00% 0
> 10.140.22.42 Up Normal 116.23 MB 25.00% 42535295865117307932921825928971026432
> 10.140.22.28 Up Normal 119.93 MB 25.00% 85070591730234615865843651857942052864
> 10.140.22.25 Up Normal 116.21 MB 25.00% 127605887595351923798765477786913079296
> [root@localhost bin]# ./nodetool -h 10.140.22.66 netstats
> Mode: Leaving: streaming data to other nodes
> Streaming to: /10.140.22.42
> /var/lib/cassandra/data/Keyspace1/Standard1-f-1-Data.db/(0,120929157)
> progress=120929157/120929157 - 100%
> /var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291)
> progress=0/3361291 - 0%
> Not receiving any streams.
> Pool Name Active Pending Completed
> Commands n/a 0 17
> Responses n/a 0 108109
> [root@usnynyc1cass02 bin]# ./nodetool -h 10.140.22.42 netstats
> Mode: Normal
> Not sending any streams.
> Streaming from: /10.140.22.66
> Keyspace1: /var/lib/cassandra/data/Keyspace1/Standard1-f-2-Data.db/(0,3361291)
> progress=0/3361291 - 0%
> Pool Name Active Pending Completed
> Commands n/a 0 11
> Responses n/a 0 107879
>
>
> Regards,
> Baskar