You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Thomas Whiteway <Th...@metaswitch.com> on 2015/05/27 14:48:02 UTC
Cassandra seems to replace existing node without specifying
replace_address
Hi,
I've been investigating using replace_address to replace a node that hasn't left the cluster cleanly and after upgrading from 2.1.0 to 2.1.4 it seems that adding a new node will automatically replace an existing node with the same IP address even if replace_address isn't used. Does anyone know whether this is an expected change (as far as I can tell it doesn't seem to be)?
I've been hitting this by
* creating a two node cluster (with the nodes in different datacentres)
* blocking traffic between the two nodes
* decommissioning one of them
* unblocking traffic between the two nodes
* trying to re-add the decommissioned node back into the cluster.
I reproduced this with some extra logging and it seems this change was introduced by CASSANDRA-7939 in 2.1.1. When StorageService.checkForEndpointCollision calls Gossiper.isFatClient TokenMetadata.tokenToEndpointMap is empty so isFatClient returns true.
Thanks,
Thomas
RE: Cassandra seems to replace existing node without specifying
replace_address
Posted by Thomas Whiteway <Th...@metaswitch.com>.
auto_bootstrap is true (we don’t set it explicitly and that’s the default). We’re using vnodes and initial_token is explicitly set to null.
Thomas
From: Robert Coli [mailto:rcoli@eventbrite.com]
Sent: 28 May 2015 19:07
To: user@cassandra.apache.org
Subject: Re: Cassandra seems to replace existing node without specifying replace_address
On Thu, May 28, 2015 at 2:00 AM, Thomas Whiteway <Th...@metaswitch.com>> wrote:
Sorry, I should have been clearer. In this case we’ve decommissioned the node and deleted the data, commitlog, and saved caches directories so we’re not hitting CASSANDRA-8801. We also hit the “A node with address <address> already exists, cancelling join” error when performing the same steps on 2.1.0, just not in 2.1.4.
Is auto_bootstrap set to true or false? Are you using vnodes, or is initial_token populated?
=Rob
Re: Cassandra seems to replace existing node without specifying replace_address
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, May 28, 2015 at 2:00 AM, Thomas Whiteway <
Thomas.Whiteway@metaswitch.com> wrote:
> Sorry, I should have been clearer. In this case we’ve decommissioned
> the node and deleted the data, commitlog, and saved caches directories so
> we’re not hitting CASSANDRA-8801. We also hit the “A node with address
> <address> already exists, cancelling join” error when performing the same
> steps on 2.1.0, just not in 2.1.4.
>
Is auto_bootstrap set to true or false? Are you using vnodes, or is
initial_token populated?
=Rob
RE: Cassandra seems to replace existing node without specifying
replace_address
Posted by Thomas Whiteway <Th...@metaswitch.com>.
Sorry, I should have been clearer. In this case we’ve decommissioned the node and deleted the data, commitlog, and saved caches directories so we’re not hitting CASSANDRA-8801. We also hit the “A node with address <address> already exists, cancelling join” error when performing the same steps on 2.1.0, just not in 2.1.4.
Thomas
From: Robert Coli [mailto:rcoli@eventbrite.com]
Sent: 27 May 2015 20:41
To: user@cassandra.apache.org
Subject: Re: Cassandra seems to replace existing node without specifying replace_address
On Wed, May 27, 2015 at 5:48 AM, Thomas Whiteway <Th...@metaswitch.com>> wrote:
I’ve been investigating using replace_address to replace a node that hasn’t left the cluster cleanly and after upgrading from 2.1.0 to 2.1.4 it seems that adding a new node will automatically replace an existing node with the same IP address even if replace_address isn’t used. Does anyone know whether this is an expected change (as far as I can tell it doesn’t seem to be)?
This is a longstanding known issue ("Cassandra has had this behavior since the inception of decom"), with a fix recently (May 19, 2015) merged to trunk.
https://issues.apache.org/jira/browse/CASSANDRA-8801
The basic problem is that the node does not forget its own cluster membership information, and so joins the cluster using its stored tokens. In my opinion, "decommission" should wipe all stored node state, but 8801 creates a workaround that addresses this, the worst case.
=Rob
Re: Cassandra seems to replace existing node without specifying replace_address
Posted by Robert Coli <rc...@eventbrite.com>.
On Wed, May 27, 2015 at 5:48 AM, Thomas Whiteway <
Thomas.Whiteway@metaswitch.com> wrote:
> I’ve been investigating using replace_address to replace a node that
> hasn’t left the cluster cleanly and after upgrading from 2.1.0 to 2.1.4 it
> seems that adding a new node will automatically replace an existing node
> with the same IP address even if replace_address isn’t used. Does anyone
> know whether this is an expected change (as far as I can tell it doesn’t
> seem to be)?
>
This is a longstanding known issue ("Cassandra has had this behavior since
the inception of decom"), with a fix recently (May 19, 2015) merged to
trunk.
https://issues.apache.org/jira/browse/CASSANDRA-8801
The basic problem is that the node does not forget its own cluster
membership information, and so joins the cluster using its stored tokens.
In my opinion, "decommission" should wipe all stored node state, but 8801
creates a workaround that addresses this, the worst case.
=Rob