when a node is dead in Cassandra cluster

["Shenghua(Daniel) Wan" <wa...@gmail.com>]

Hi,
When a node is dead, is it supposed to exist in the ring? When I found a
node is lost, and I check with nodetool and ops center, I still see the
lost node in the token ring. When I describe_ring, the lost node is also
returned. Is this what it is supposed to be? Why did not C* server hide the
lost nodes from the clients?

Thanks a lot!

-- 

Regards,
Shenghua (Daniel) Wan

Re: when a node is dead in Cassandra cluster

[John Wong <go...@gmail.com>]

On Mon, Sep 21, 2015 at 8:32 PM, Shenghua(Daniel) Wan <wanshenghua@gmail.com
> wrote:

> Hi,
> When a node is dead, is it supposed to exist in the ring?
>

It is still considered as part of a cluster. Imagine a case when you do a
rolling restart, the node would be temporary out of service for maybe a few
minutes to a few hours and it would be DN in the status column.

Why did not C* server hide the lost nodes from the clients?


Is it the client contacting this dead node directly, or is it the Cassandra
coordinator unable to get a quorum?

Re: when a node is dead in Cassandra cluster

[Jeff Ferland <jb...@tubularlabs.com>]

A dead node should exist in the ring until it is replaced. If you remove a node without a replacement, you’ll end up with that replica’s ownership being placed onto another node without the data having been transferred, and queries against that range will falsely empty records until a repair is completed. I believe the most correct action is to start up a replacement node using the -Dcassandra.replace_address=ip_of_dead_node command line argument. Reference http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html <http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html>.

-Jeff

> On Sep 21, 2015, at 5:32 PM, Shenghua(Daniel) Wan <wa...@gmail.com> wrote:
> 
> Hi, 
> When a node is dead, is it supposed to exist in the ring? When I found a node is lost, and I check with nodetool and ops center, I still see the lost node in the token ring. When I describe_ring, the lost node is also returned. Is this what it is supposed to be? Why did not C* server hide the lost nodes from the clients?
> 
> Thanks a lot!
> 
> -- 
> 
> Regards,
> Shenghua (Daniel) Wan