Decommission a node causing high IO usage on 2 other nodes

[huyle <hu...@springpartners.com>]

Hi,

We recently upgraded a cluster to larger servers by bootstrapping the new
servers using token - 1 from old servers as the tokens for new servers.
After bootstrapping the new servers, old servers are shown owning 0.00%
workload ( they still really own a very small amount of data as I understand
it).   After decommissioning one of the old node, 2 new nodes next in the
token ring got IO wait from normal of 3% to 60%+.  Then we ran compaction on
the 2 new nodes that got high IO wait, IO is at 30%.  That 10 times the IO
wait we had before.  

Here is snippet of token ring for illustration purpose:

1.new
1.old
2.new
2.old
3.new
3.old
etc...

after decommission 1.old, 2.new and 3.new got very high IO wait.

What may be causing high IO wait on 2.new and 3.new?

Thanks!

Huy


--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Decommission-a-node-causing-high-IO-usage-on-2-other-nodes-tp7044866p7044866.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: Decommission a node causing high IO usage on 2 other nodes

[huyle <hu...@springpartners.com>]

Yes, we do have RF=3.  

That explains it all.  Thanks Peter!

Huy


Peter Schuller wrote
> 
>> What may be causing high IO wait on 2.new and 3.new?
> 
> Do you have RF=3?
> 
> The problem with 'nodetool ring' in terms of interpreting those '0%'
> is that it does not take RF and replication strategy into account. If
> you have RF=3, whatever data has its primary range assigned to node N
> will also be on N+1 and N+2. Additionally, whatever data has it's
> primary range assigned to N-1 and N-2 will also have a copy on N.
> 
> So despite all the old nodes showing up as 0% in nodetool ring, they
> are in fact responsible for data. And you would expect to see nodes
> N+2 and N-2 be the two "old" nodes affected by decommissioning node N.
> (Unless I'm tripping myself up somewhere now...)
> 
> -- 
> / Peter Schuller (@scode, http://worldmodscode.wordpress.com)
> 


--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Decommission-a-node-causing-high-IO-usage-on-2-other-nodes-tp7044866p7045589.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Re: Decommission a node causing high IO usage on 2 other nodes

[Peter Schuller <pe...@infidyne.com>]

> What may be causing high IO wait on 2.new and 3.new?

Do you have RF=3?

The problem with 'nodetool ring' in terms of interpreting those '0%'
is that it does not take RF and replication strategy into account. If
you have RF=3, whatever data has its primary range assigned to node N
will also be on N+1 and N+2. Additionally, whatever data has it's
primary range assigned to N-1 and N-2 will also have a copy on N.

So despite all the old nodes showing up as 0% in nodetool ring, they
are in fact responsible for data. And you would expect to see nodes
N+2 and N-2 be the two "old" nodes affected by decommissioning node N.
(Unless I'm tripping myself up somewhere now...)

-- 
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)