You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jai Bheemsen Rao Dhanwada <ja...@gmail.com> on 2017/08/30 23:06:14 UTC
Cassandra snapshot restore with VNODES missing some data
Hello All,
I am trying to restore a cluster with VNODE(s) to a new cluster using the
snapshot.
After the restore when I query data from cql I see some random data is
missing.
I used the below steps to restore
1. Snapshot on the source cluster
2. Setup new cluster(VNODEs) with the same schema as source cluster.
3. Restore the snapshot to the specific CF directory
4. Restart the node.
5. Now when I query through CQL I see some data is missing.
I looked at the procedure documented in :
http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_snapshot_restore_new_cluster.html
When I try to follow the steps in the above doc, especially replacing the
num_tokens with initial_token: from the source cluster token, I get the
below error.
ERROR [main] 2017-08-30 17:17:20,878 CassandraDaemon.java:395 - Fatal
configuration error org.apache.cassandra.exceptions.ConfigurationException:
Cannot change the number of tokens from 256 to 1 at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:972)
~[apache-cassandra-2.1.16.jar:2.1.16] at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
~[apache-cassandra-2.1.16.jar:2.1.16] at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
~[apache-cassandra-2.1.16.jar:2.1.16] at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
[apache-cassandra-2.1.16.jar:2.1.16] at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
[apache-cassandra-2.1.16.jar:2.1.16] at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
[apache-cassandra-2.1.16.jar:2.1.16] INFO [StorageServiceShutdownHook]
2017-08-30 17:17:20,898 Gossiper.java:1454 - Announcing shutdown INFO
[StorageServiceShutdownHook] 2017-08-30 17:17:22,902
MessagingService.java:734 - Waiting for messaging service to quiesce INFO
[ACCEPT-/x <http://10.16.4.182/>.x.x.x] 2017-08-30 17:17:22,903
MessagingService.java:1020 - MessagingService has terminated the accept()
thread
any pointers on this?
Re: Cassandra snapshot restore with VNODES missing some data
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
Hello Kurt,
Thanks for the help :)
On Fri, Sep 1, 2017 at 1:12 PM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
> yes looks like I am missing that.
>
> Let me test on one node and try a full cluster restore.
>
> will update here once I complete my test
>
> On Fri, Sep 1, 2017 at 5:01 AM, kurt greaves <ku...@instaclustr.com> wrote:
>
>> is num_tokens also set to 256?
>>
>
>
Re: Cassandra snapshot restore with VNODES missing some data
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
yes looks like I am missing that.
Let me test on one node and try a full cluster restore.
will update here once I complete my test
On Fri, Sep 1, 2017 at 5:01 AM, kurt greaves <ku...@instaclustr.com> wrote:
> is num_tokens also set to 256?
>
Re: Cassandra snapshot restore with VNODES missing some data
Posted by kurt greaves <ku...@instaclustr.com>.
is num_tokens also set to 256?
Re: Cassandra snapshot restore with VNODES missing some data
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
I double checked that I am setting all 256 tokens, verified manually.
When I start Cassandra with empty data directory is startsup fine. now if I
restart the Cassandra without making any changes it won't start and give
same error
I captured the nodetool status and nodetool ring and compare the tokens
value
Datacenter: us-west-2
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN x.x.x.x 51.63 KB *256* 100.0%
d1248cf3-12cd-4011-908c-2b4c8c333920 2a
ERROR [main] 2017-08-31 15:44:51,339 CassandraDaemon.java:395 - Fatal
configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot change the
number of tokens from 256 to 1
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:972)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
~[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
[apache-cassandra-2.1.16.jar:2.1.16]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
[apache-cassandra-2.1.16.jar:2.1.16]
INFO [StorageServiceShutdownHook] 2017-08-31 15:44:51,355
Gossiper.java:1454 - Announcing shutdown
INFO [StorageServiceShutdownHook] 2017-08-31 15:44:53,360
MessagingService.java:734 - Waiting for messaging service to quiesce
INFO [ACCEPT-/x.x.x.x] 2017-08-31 15:44:53,361 MessagingService.java:1020
- MessagingService has terminated the accept() thread
On Thu, Aug 31, 2017 at 12:51 AM, kurt greaves <ku...@instaclustr.com> wrote:
> What Erick said. That error in particular implies you aren't setting all
> 256 tokens in initial_token
>
Re: Cassandra snapshot restore with VNODES missing some data
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Thu, Aug 31, 2017 at 10:14 AM, Lutaya Shafiq Holmes <
lutayashafiq@gmail.com> wrote:
> SOME ONE HELP ME GET STARTED WITH CASSANDRA IN WINDOWS
Given your user profile picture I've freaked out for a second thinking it's
the 45th president of US is shouting at us to get started with Cassandra...
Seriously, I would not expect anyone jumping in for that type of request on
an unrelated email thread. Please start a new thread and explain what
seems to be your current problem: what have you already tired and what
resulted from that (vs. what you've expected).
Cheers,
--
Alex
Re: Cassandra snapshot restore with VNODES missing some data
Posted by Lutaya Shafiq Holmes <lu...@gmail.com>.
SOME ONE HELP ME GET STARTED WITH CASSANDRA IN WINDOWS<div
id="DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2"><br />
<table style="border-top: 1px solid #D3D4DE;">
<tr>
<td style="width: 55px; padding-top: 13px;"><a
href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon"
target="_blank"><img
src="https://ipmcdn.avast.com/images/icons/icon-envelope-tick-round-orange-animated-no-repeat-v1.gif"
alt="" width="46" height="29" style="width: 46px; height: 29px;"
/></a></td>
<td style="width: 470px; padding-top: 12px; color: #41424e;
font-size: 13px; font-family: Arial, Helvetica, sans-serif;
line-height: 18px;">Virus-free. <a
href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link"
target="_blank" style="color: #4453ea;">www.avast.com</a>
</td>
</tr>
</table><a href="#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2" width="1"
height="1"></a></div>
On 8/31/17, kurt greaves <ku...@instaclustr.com> wrote:
> What Erick said. That error in particular implies you aren't setting all
> 256 tokens in initial_token
>
--
Lutaaya Shafiq
Mobile: +256702772721 | +256783564130
Twitter: @lutayashafiq
Skype: lutaya5
Home: lutayashafiq.com
http://www.fourcornersalliancegroup.com/?a=shafiqholmes
"The most beautiful people we have known are those who have known defeat,
known suffering, known struggle, known loss and have found their way out of
the depths. These persons have an appreciation, a sensitivity and an
understanding of life that fills them with compassion, gentleness and a
deep loving concern. Beautiful people do not just happen." - *Elisabeth
Kubler-Ross*
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Cassandra snapshot restore with VNODES missing some data
Posted by kurt greaves <ku...@instaclustr.com>.
What Erick said. That error in particular implies you aren't setting all
256 tokens in initial_token
Re: Cassandra snapshot restore with VNODES missing some data
Posted by Erick Ramirez <fl...@gmail.com>.
For your method to work, you have to restore like-for-like, i.e. you need
to mirror the source nodes by using the exact same tokens in system.local.
For example, if source node A has tokens 567, 678 and 789, then you need to
setup the equivalent target node with exactly those tokens. Otherwise, the
new nodes will own a different range of tokens and the data in the sstables
will not necessarily match. Cheers!
On Thu, Aug 31, 2017 at 9:06 AM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
> Hello All,
>
> I am trying to restore a cluster with VNODE(s) to a new cluster using the
> snapshot.
> After the restore when I query data from cql I see some random data is
> missing.
>
> I used the below steps to restore
>
> 1. Snapshot on the source cluster
> 2. Setup new cluster(VNODEs) with the same schema as source cluster.
> 3. Restore the snapshot to the specific CF directory
> 4. Restart the node.
> 5. Now when I query through CQL I see some data is missing.
>
> I looked at the procedure documented in : http://docs.datastax.com/en/
> cassandra/2.1/cassandra/operations/ops_snapshot_restore_new_cluster.html
>
> When I try to follow the steps in the above doc, especially replacing the
> num_tokens with initial_token: from the source cluster token, I get the
> below error.
>
> ERROR [main] 2017-08-30 17:17:20,878 CassandraDaemon.java:395 - Fatal
> configuration error org.apache.cassandra.exceptions.ConfigurationException:
> Cannot change the number of tokens from 256 to 1 at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:972)
> ~[apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
> StorageService.initServer(StorageService.java:740)
> ~[apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
> StorageService.initServer(StorageService.java:617)
> ~[apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
> CassandraDaemon.setup(CassandraDaemon.java:391)
> [apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
> CassandraDaemon.activate(CassandraDaemon.java:566)
> [apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
> CassandraDaemon.main(CassandraDaemon.java:655)
> [apache-cassandra-2.1.16.jar:2.1.16] INFO [StorageServiceShutdownHook]
> 2017-08-30 17:17:20,898 Gossiper.java:1454 - Announcing shutdown INFO
> [StorageServiceShutdownHook] 2017-08-30 17:17:22,902
> MessagingService.java:734 - Waiting for messaging service to quiesce INFO
> [ACCEPT-/x <http://10.16.4.182/>.x.x.x] 2017-08-30 17:17:22,903
> MessagingService.java:1020 - MessagingService has terminated the accept()
> thread
>
> any pointers on this?
>
Re: Cassandra snapshot restore with VNODES missing some data
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
Also,
When use the same DC and Rack, I am still getting the below error when I
specify the 256 tokens from the source cluster and restart it.
ERROR [main] 2017-08-30 17:17:20,878 CassandraDaemon.java:395 - Fatal
configuration error org.apache.cassandra.exceptions.ConfigurationException:
Cannot change the number of tokens from 256 to 1 at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:972)
~[apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
StorageService.initServer(StorageService.java:740)
~[apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
StorageService.initServer(StorageService.java:617)
~[apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
CassandraDaemon.setup(CassandraDaemon.java:391)
[apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
CassandraDaemon.activate(CassandraDaemon.java:566)
[apache-cassandra-2.1.16.jar:2.1.16] at org.apache.cassandra.service.
CassandraDaemon.main(CassandraDaemon.java:655)
[apache-cassandra-2.1.16.jar:2.1.16]
INFO [StorageServiceShutdownHook] 2017-08-30 17:17:20,898
Gossiper.java:1454 - Announcing shutdown INFO [StorageServiceShutdownHook]
2017-08-30 17:17:22,902 MessagingService.java:734 - Waiting for messaging
service to quiesce INFO [ACCEPT-/x <http://10.16.4.182/>.x.x.x] 2017-08-30
17:17:22,903 MessagingService.java:1020 - MessagingService has terminated
the accept() thread
On Wed, Aug 30, 2017 at 5:57 PM, Jai Bheemsen Rao Dhanwada <
jaibheemsen@gmail.com> wrote:
> yes source use the vnodes.
>
> I am restoring to a different cluster in different datacenter, so the rack
> and dc changes. does that matter?
>
> On Wed, Aug 30, 2017 at 5:55 PM, kurt greaves <ku...@instaclustr.com>
> wrote:
>
>> Does the source cluster also use vnodes? You will need to ensure you use
>> the same tokens for each node as the snapshots used in the source (and also
>> ensure same tokens apply to same racks).
>>
>
>
Re: Cassandra snapshot restore with VNODES missing some data
Posted by Jai Bheemsen Rao Dhanwada <ja...@gmail.com>.
yes source use the vnodes.
I am restoring to a different cluster in different datacenter, so the rack
and dc changes. does that matter?
On Wed, Aug 30, 2017 at 5:55 PM, kurt greaves <ku...@instaclustr.com> wrote:
> Does the source cluster also use vnodes? You will need to ensure you use
> the same tokens for each node as the snapshots used in the source (and also
> ensure same tokens apply to same racks).
>
Re: Cassandra snapshot restore with VNODES missing some data
Posted by kurt greaves <ku...@instaclustr.com>.
Does the source cluster also use vnodes? You will need to ensure you use
the same tokens for each node as the snapshots used in the source (and also
ensure same tokens apply to same racks).