Nodetool removenode => Stream stuck

[Philippe Dupont <pd...@teads.tv>]

Hi guys,

Recently a node in our ring became unreachable due to some AWS EC2 issue,
and we decided to remove it using "nodetool removenode" command.
As we are using Vnodes on a 28 node cluster, the removenode command
generates a lot of streams between all nodes. The problem is that some
stream becomes stuck.
It's the third time we use the removenode command and each time, some
stream was stuck

The solution we found is to force the removenode (nodetool removenode
force) and then restart the nodes with stream pending to get rid of them

Is there an explanation of this issue?

Thanks,

Philippe

Re: Nodetool removenode => Stream stuck

[Robert Coli <rc...@eventbrite.com>]

On Thu, Jan 2, 2014 at 7:54 AM, Philippe Dupont <pd...@teads.tv> wrote:

> As we are using Vnodes on a 28 node cluster, the removenode command
> generates a lot of streams between all nodes. The problem is that some
> stream becomes stuck.
> It's the third time we use the removenode command and each time, some
> stream was stuck
>

What version of Cassandra? If 1.2.x, streaming has been rewritten
significantly in 2.0, and so in theory should be less likely to do this.
Note that I do not recommend upgrading to 2.0 in production at this time.


> The solution we found is to force the removenode (nodetool removenode
> force) and then restart the nodes with stream pending to get rid of them
>

The solution you found means that the streaming part of removenode was
interrupted, which means you should repair ASAP.

https://issues.apache.org/jira/browse/CASSANDRA-3486

Is the ticket where your input as to the desirability of a facility to
stop/restart streaming operations without restarting nodes is welcome.

=Rob