You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Arindam Barua <ab...@247-inc.com> on 2013/06/25 08:19:08 UTC
Problems with node rejoining cluster
We need to do a rolling upgrade of our Cassandra cluster in production, since we are upgrading Cassandra on solaris to Cassandra on CentOS.
(We went with solaris initially since most of our other hosts in production are solaris, but were running into some lockup issues during perf tests, and decided to switch to linux)
Here are the steps we are following to take the node out of service, and get it back. Can someone comment if we are missing anything (eg. is it recommended to specify tokens in cassandra.yaml, or do something different with the seed hosts than mentioned below)
1. nodetool decommission - wait for the data to be streamed out.
2. Re-image (everything is wiped off the disks) the host to CentOS, with the same Cassandra version
3. Get Cassandra back up.
Other details:
- Using Cassandra 1.1.5
- We do not specify any tokens in cassandra.yaml relying on bootstrap assigning the tokens automatically.
- We are testing with a 4 node cluster, with only one seed host. The seed host is specified in the cassandra.yaml of each node and is not changed at any point.
While testing the solaris to linux upgrade path, things seem to work smoothly. The data streams out fine, and streams back in when the node comes back up. However, testing the linux to solaris path (in case we need to rollback), we are facing some issues with the nodes joining back the ring. nodetool indicates that the node has joined back the ring, but no data streams in, the node doesn't know about the keyspaces/column families, etc. We see some errors in the logs of the newly added nodes pasted below.
[17/06/2013:14:10:17 PDT] MutationStage:1: ERROR RowMutationVerbHandler.java (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1020
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
Thanks,
Arindam
Re: Problems with node rejoining cluster
Posted by aaron morton <aa...@thelastpickle.com>.
> When you say copy all the data and config in step 2 below, is that for backup reasons, or do I need to actually copy it back manually to the box?
That was to move to another box, but you can use the same process to re-build the box. Just get the data and config off the node then put it back.
> nodetool always reports it joins back the cluster fine. cassandra-cli does not report split schema.
mmm, somehow the node got a mutation for a CF it does not have.
Does the error happen all the time.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 28/06/2013, at 8:10 PM, Arindam Barua <ab...@247-inc.com> wrote:
>
> I am re-installing on the same machine (the IP stays the same), but need to wipe all the disks for the OS switch.
>
> When you say copy all the data and config in step 2 below, is that for backup reasons, or do I need to actually copy it back manually to the box? When the node joins back the cluster, should it not automatically get the data via bootstrapping?
>
> >> The error you got says that the schema was not replicated. Check that the node is part of cluster and check for a split schema using cassandra-cli (FAQ on the wiki has help for split schema)
> nodetool always reports it joins back the cluster fine. cassandra-cli does not report split schema.
>
> Thanks,
> Arindam
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: Thursday, June 27, 2013 10:11 PM
> To: user@cassandra.apache.org
> Subject: Re: Problems with node rejoining cluster
>
> Are you move the node to a new machine or re-installing on the same machine ?
>
> If it's the former then:
> * shut it down cleanly
> * copy all the data and config
> * update the yaml with the new IP for list_address, rpc_address and seed_list
> * restart the node
>
> The error you got says that the schema was not replicated. Check that the node is part of cluster and check for a split schema using cassandra-cli (FAQ on the wiki has help for split schema)
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 27/06/2013, at 8:57 AM, Arindam Barua <ab...@247-inc.com> wrote:
>
>
>
> Thanks for your response.
>
> Are there any other general comments on the steps we are taking to decommission and join back the node. I'm assuming if we do specify a token, we should specify exactly the same token when we add that node back.
>
> -----Original Message-----
> From: Robert Coli [mailto:rcoli@eventbrite.com]
> Sent: Tuesday, June 25, 2013 11:15 AM
> To: user@cassandra.apache.org
> Subject: Re: Problems with node rejoining cluster
>
> On Mon, Jun 24, 2013 at 11:19 PM, Arindam Barua <ab...@247-inc.com> wrote:
>
> - We do not specify any tokens in cassandra.yaml relying on
> bootstrap assigning the tokens automatically.
>
> As cassandra.yaml comments state, you should never ever do this in a real cluster.
>
> I don't know what is causing your underlying issue, but not-specifying tokens is a strong contender.
>
> =Rob
>
>
RE: Problems with node rejoining cluster
Posted by Arindam Barua <ab...@247-inc.com>.
I am re-installing on the same machine (the IP stays the same), but need to wipe all the disks for the OS switch.
When you say copy all the data and config in step 2 below, is that for backup reasons, or do I need to actually copy it back manually to the box? When the node joins back the cluster, should it not automatically get the data via bootstrapping?
>> The error you got says that the schema was not replicated. Check that the node is part of cluster and check for a split schema using cassandra-cli (FAQ on the wiki has help for split schema)
nodetool always reports it joins back the cluster fine. cassandra-cli does not report split schema.
Thanks,
Arindam
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Thursday, June 27, 2013 10:11 PM
To: user@cassandra.apache.org
Subject: Re: Problems with node rejoining cluster
Are you move the node to a new machine or re-installing on the same machine ?
If it's the former then:
* shut it down cleanly
* copy all the data and config
* update the yaml with the new IP for list_address, rpc_address and seed_list
* restart the node
The error you got says that the schema was not replicated. Check that the node is part of cluster and check for a split schema using cassandra-cli (FAQ on the wiki has help for split schema)
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 27/06/2013, at 8:57 AM, Arindam Barua <ab...@247-inc.com>> wrote:
Thanks for your response.
Are there any other general comments on the steps we are taking to decommission and join back the node. I'm assuming if we do specify a token, we should specify exactly the same token when we add that node back.
-----Original Message-----
From: Robert Coli [mailto:rcoli@eventbrite.com<http://eventbrite.com>]
Sent: Tuesday, June 25, 2013 11:15 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Problems with node rejoining cluster
On Mon, Jun 24, 2013 at 11:19 PM, Arindam Barua <ab...@247-inc.com>> wrote:
- We do not specify any tokens in cassandra.yaml relying on
bootstrap assigning the tokens automatically.
As cassandra.yaml comments state, you should never ever do this in a real cluster.
I don't know what is causing your underlying issue, but not-specifying tokens is a strong contender.
=Rob
Re: Problems with node rejoining cluster
Posted by aaron morton <aa...@thelastpickle.com>.
Are you move the node to a new machine or re-installing on the same machine ?
If it's the former then:
* shut it down cleanly
* copy all the data and config
* update the yaml with the new IP for list_address, rpc_address and seed_list
* restart the node
The error you got says that the schema was not replicated. Check that the node is part of cluster and check for a split schema using cassandra-cli (FAQ on the wiki has help for split schema)
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 27/06/2013, at 8:57 AM, Arindam Barua <ab...@247-inc.com> wrote:
>
> Thanks for your response.
>
> Are there any other general comments on the steps we are taking to decommission and join back the node. I'm assuming if we do specify a token, we should specify exactly the same token when we add that node back.
>
> -----Original Message-----
> From: Robert Coli [mailto:rcoli@eventbrite.com]
> Sent: Tuesday, June 25, 2013 11:15 AM
> To: user@cassandra.apache.org
> Subject: Re: Problems with node rejoining cluster
>
> On Mon, Jun 24, 2013 at 11:19 PM, Arindam Barua <ab...@247-inc.com> wrote:
>> - We do not specify any tokens in cassandra.yaml relying on
>> bootstrap assigning the tokens automatically.
>
> As cassandra.yaml comments state, you should never ever do this in a real cluster.
>
> I don't know what is causing your underlying issue, but not-specifying tokens is a strong contender.
>
> =Rob
>
>
RE: Problems with node rejoining cluster
Posted by Arindam Barua <ab...@247-inc.com>.
Thanks for your response.
Are there any other general comments on the steps we are taking to decommission and join back the node. I'm assuming if we do specify a token, we should specify exactly the same token when we add that node back.
-----Original Message-----
From: Robert Coli [mailto:rcoli@eventbrite.com]
Sent: Tuesday, June 25, 2013 11:15 AM
To: user@cassandra.apache.org
Subject: Re: Problems with node rejoining cluster
On Mon, Jun 24, 2013 at 11:19 PM, Arindam Barua <ab...@247-inc.com> wrote:
> - We do not specify any tokens in cassandra.yaml relying on
> bootstrap assigning the tokens automatically.
As cassandra.yaml comments state, you should never ever do this in a real cluster.
I don't know what is causing your underlying issue, but not-specifying tokens is a strong contender.
=Rob
Re: Problems with node rejoining cluster
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Jun 24, 2013 at 11:19 PM, Arindam Barua <ab...@247-inc.com> wrote:
> - We do not specify any tokens in cassandra.yaml relying on
> bootstrap assigning the tokens automatically.
As cassandra.yaml comments state, you should never ever do this in a
real cluster.
I don't know what is causing your underlying issue, but not-specifying
tokens is a strong contender.
=Rob