You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Baskar Duraikannu <ba...@gmail.com> on 2013/07/09 02:45:17 UTC
Node tokens / data move
We have two clusters used by two different groups with vnodes enabled. Now there is a need to move some of the keyspaces from cluster 1 to cluster 2.
Can I just copy data files for the required keyspaces, create schema manually and run repair?
Anything else required? Please help.
--
Thanks,
Baskar Duraikannu
Re: Node tokens / data move
Posted by Jeremiah D Jordan <je...@gmail.com>.
Pretty sure you can put the list in the yaml file too.
-Jeremiah
On Jul 12, 2013, at 3:09 AM, aaron morton <aa...@thelastpickle.com> wrote:
>> Can he not specify all 256 tokens in the YAML of the new cluster and then copy sstables?
>> I know it is a bit ugly but should work.
> You can pass a comma separated list of tokens to the -Dcassandra.replace_token JVM param.
>
> AFAIK it's not possible to provide the list in the yaml file.
>
> Cheers
> A
>
> -----------------
> Aaron Morton
> Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 11/07/2013, at 5:07 AM, Baskar Duraikannu <ba...@gmail.com> wrote:
>
>>
>> I copied the sstables and then ran a repair. It worked. Looks like export and import may have been much faster given that we had very little data.
>>
>> Thanks everyone.
>>
>>
>>
>>
>> On Tue, Jul 9, 2013 at 1:34 PM, sankalp kohli <ko...@gmail.com> wrote:
>> Hi Aaron,
>> Can he not specify all 256 tokens in the YAML of the new cluster and then copy sstables?
>> I know it is a bit ugly but should work.
>>
>> Sankalp
>>
>>
>> On Tue, Jul 9, 2013 at 3:19 AM, Baskar Duraikannu <ba...@gmail.com> wrote:
>> Thanks Aaron
>>
>> On 7/9/13, aaron morton <aa...@thelastpickle.com> wrote:
>> >> Can I just copy data files for the required keyspaces, create schema
>> >> manually and run repair?
>> > If you have something like RF 3 and 3 nodes then yes, you can copy the data
>> > from one node in the source cluster to all nodes in the dest cluster and use
>> > cleanup to remove the unneeded data. Because each node in the source cluster
>> > has a full copy of the data.
>> >
>> > If that's not the case you cannot copy the data files, even if they have the
>> > same number of nodes, because the nodes in the dest cluster will have
>> > different tokens. AFAIK you need to export the full data set from the source
>> > DC and then import it into the dest system.
>> >
>> > The Bulk Load utility may be of help
>> > http://www.datastax.com/docs/1.2/references/bulkloader . You could copy the
>> > SSTables from every node in the source system and bulk load them into the
>> > dest system. That process will ensure rows are sent to nodes that are
>> > replicas.
>> >
>> > Cheers
>> >
>> > -----------------
>> > Aaron Morton
>> > Freelance Cassandra Consultant
>> > New Zealand
>> >
>> > @aaronmorton
>> > http://www.thelastpickle.com
>> >
>> > On 9/07/2013, at 12:45 PM, Baskar Duraikannu
>> > <ba...@gmail.com> wrote:
>> >
>> >> We have two clusters used by two different groups with vnodes enabled. Now
>> >> there is a need to move some of the keyspaces from cluster 1 to cluster 2.
>> >>
>> >>
>> >> Can I just copy data files for the required keyspaces, create schema
>> >> manually and run repair?
>> >>
>> >> Anything else required? Please help.
>> >> --
>> >> Thanks,
>> >> Baskar Duraikannu
>> >
>> >
>>
>>
>
Re: Node tokens / data move
Posted by Eric Stevens <mi...@gmail.com>.
>
> if you've got a non-vnode cluster and are trying to convert, you are
> likely going to at least want, if not have to, run shuffle
Fair enough. Running shuffle after upgrading to using vnodes is nearly
mandatory or else you'll run into troubles when adding more nodes (see this
Jira ticket<https://issues.apache.org/jira/browse/CASSANDRA-5525?focusedCommentId=13645445&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13645445>).
This should be a one-time occurrence during the upgrade process, and
hopefully you can do it off-hours.
On Tue, Jul 16, 2013 at 9:09 AM, David McNelis <dm...@gmail.com> wrote:
> Eric,
>
> Unfortunately if you've got a non-vnode cluster and are trying to convert,
> you are likely going to at least want, if not have to, run shuffle. It
> isn't a pleasant situation when you run into that because in order for the
> shuffle to execute safely and successfully you need to have essentially
> 2x's the disk space as the size of your data at a minimum because no nodes
> are removed from the original node until a cleanup is called.
>
> This also means that other cass operations that cause disk usage to ballon
> are competing for space with the shuffle process and boom, like that, out
> of disk space.
>
> David
>
>
> On Tue, Jul 16, 2013 at 8:35 AM, Eric Stevens <mi...@gmail.com> wrote:
>
>> vnodes currently do not brings any noticeable benefits to outweight
>>> trouble
>>
>>
>> The main advantage of vnodes is that it lets you have flexibility with
>> respect to adding and removing nodes from your cluster without having to
>> rebalance your cluster (issuing a lot of moves). A shuffle is a lot of
>> moves too, so the only reason to do this is if you've discovered that the
>> random allocation of vnodes has managed to leave an unbalanced cluster (one
>> node either has an unusually large load, or an unusually small one).
>>
>> Shuffling isn't something you should be doing as a matter of course, and
>> with 256 vnodes per server and 3+ servers, the chances of a brokenly
>> unbalanced cluster is more likely the cause of bad data partitioning
>> (extremely unbalanced row sizes) than of vnode token placement. With
>> unbalanced rows, nothing will solve balance problems except finding a
>> different way to partition your data across rows.
>>
>>
>>
>> On Mon, Jul 15, 2013 at 7:10 AM, Radim Kolar <hs...@filez.com> wrote:
>>
>>>
>>> My understanding is that it is not possible to change the number of
>>>> tokens after the node has been initialized.
>>>>
>>> that was my conclusion too. vnodes currently do not brings any
>>> noticeable benefits to outweight trouble. shuffle is very slow in large
>>> cluster. Recovery is faster with vnodes but i have very few node failures
>>> per year.
>>>
>>
>>
>
Re: Node tokens / data move
Posted by David McNelis <dm...@gmail.com>.
Eric,
Unfortunately if you've got a non-vnode cluster and are trying to convert,
you are likely going to at least want, if not have to, run shuffle. It
isn't a pleasant situation when you run into that because in order for the
shuffle to execute safely and successfully you need to have essentially
2x's the disk space as the size of your data at a minimum because no nodes
are removed from the original node until a cleanup is called.
This also means that other cass operations that cause disk usage to ballon
are competing for space with the shuffle process and boom, like that, out
of disk space.
David
On Tue, Jul 16, 2013 at 8:35 AM, Eric Stevens <mi...@gmail.com> wrote:
> vnodes currently do not brings any noticeable benefits to outweight trouble
>
>
> The main advantage of vnodes is that it lets you have flexibility with
> respect to adding and removing nodes from your cluster without having to
> rebalance your cluster (issuing a lot of moves). A shuffle is a lot of
> moves too, so the only reason to do this is if you've discovered that the
> random allocation of vnodes has managed to leave an unbalanced cluster (one
> node either has an unusually large load, or an unusually small one).
>
> Shuffling isn't something you should be doing as a matter of course, and
> with 256 vnodes per server and 3+ servers, the chances of a brokenly
> unbalanced cluster is more likely the cause of bad data partitioning
> (extremely unbalanced row sizes) than of vnode token placement. With
> unbalanced rows, nothing will solve balance problems except finding a
> different way to partition your data across rows.
>
>
>
> On Mon, Jul 15, 2013 at 7:10 AM, Radim Kolar <hs...@filez.com> wrote:
>
>>
>> My understanding is that it is not possible to change the number of
>>> tokens after the node has been initialized.
>>>
>> that was my conclusion too. vnodes currently do not brings any noticeable
>> benefits to outweight trouble. shuffle is very slow in large cluster.
>> Recovery is faster with vnodes but i have very few node failures per year.
>>
>
>
Re: Node tokens / data move
Posted by Eric Stevens <mi...@gmail.com>.
>
> vnodes currently do not brings any noticeable benefits to outweight trouble
The main advantage of vnodes is that it lets you have flexibility with
respect to adding and removing nodes from your cluster without having to
rebalance your cluster (issuing a lot of moves). A shuffle is a lot of
moves too, so the only reason to do this is if you've discovered that the
random allocation of vnodes has managed to leave an unbalanced cluster (one
node either has an unusually large load, or an unusually small one).
Shuffling isn't something you should be doing as a matter of course, and
with 256 vnodes per server and 3+ servers, the chances of a brokenly
unbalanced cluster is more likely the cause of bad data partitioning
(extremely unbalanced row sizes) than of vnode token placement. With
unbalanced rows, nothing will solve balance problems except finding a
different way to partition your data across rows.
On Mon, Jul 15, 2013 at 7:10 AM, Radim Kolar <hs...@filez.com> wrote:
>
> My understanding is that it is not possible to change the number of
>> tokens after the node has been initialized.
>>
> that was my conclusion too. vnodes currently do not brings any noticeable
> benefits to outweight trouble. shuffle is very slow in large cluster.
> Recovery is faster with vnodes but i have very few node failures per year.
>
Re: Node tokens / data move
Posted by Radim Kolar <hs...@filez.com>.
> My understanding is that it is not possible to change the number of
> tokens after the node has been initialized.
that was my conclusion too. vnodes currently do not brings any
noticeable benefits to outweight trouble. shuffle is very slow in large
cluster. Recovery is faster with vnodes but i have very few node
failures per year.
Re: Node tokens / data move
Posted by aaron morton <aa...@thelastpickle.com>.
> Pretty sure you can put the list in the yaml file too.
Yup, sorry.
initial_tokens can take a comma separated value
Cheers
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 15/07/2013, at 9:44 AM, Eric Stevens <mi...@gmail.com> wrote:
> My understanding is that it is not possible to change the number of tokens after the node has been initialized. To do so you would first need to decommission the node, then start it clean with the appropriate num_tokens in the yaml.
>
>
> On Fri, Jul 12, 2013 at 9:17 PM, Radim Kolar <hs...@filez.com> wrote:
> its possible to change num_tokens on node with data?
>
> i changed it and restarted node but it still has same amount in nodetool status.
>
Re: Node tokens / data move
Posted by Eric Stevens <mi...@gmail.com>.
My understanding is that it is not possible to change the number of tokens
after the node has been initialized. To do so you would first need to
decommission the node, then start it clean with the appropriate num_tokens
in the yaml.
On Fri, Jul 12, 2013 at 9:17 PM, Radim Kolar <hs...@filez.com> wrote:
> its possible to change num_tokens on node with data?
>
> i changed it and restarted node but it still has same amount in nodetool
> status.
>
Re: Node tokens / data move
Posted by Radim Kolar <hs...@filez.com>.
its possible to change num_tokens on node with data?
i changed it and restarted node but it still has same amount in nodetool
status.
Re: Node tokens / data move
Posted by aaron morton <aa...@thelastpickle.com>.
> Can he not specify all 256 tokens in the YAML of the new cluster and then copy sstables?
> I know it is a bit ugly but should work.
You can pass a comma separated list of tokens to the -Dcassandra.replace_token JVM param.
AFAIK it's not possible to provide the list in the yaml file.
Cheers
A
-----------------
Aaron Morton
Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 11/07/2013, at 5:07 AM, Baskar Duraikannu <ba...@gmail.com> wrote:
>
> I copied the sstables and then ran a repair. It worked. Looks like export and import may have been much faster given that we had very little data.
>
> Thanks everyone.
>
>
>
>
> On Tue, Jul 9, 2013 at 1:34 PM, sankalp kohli <ko...@gmail.com> wrote:
> Hi Aaron,
> Can he not specify all 256 tokens in the YAML of the new cluster and then copy sstables?
> I know it is a bit ugly but should work.
>
> Sankalp
>
>
> On Tue, Jul 9, 2013 at 3:19 AM, Baskar Duraikannu <ba...@gmail.com> wrote:
> Thanks Aaron
>
> On 7/9/13, aaron morton <aa...@thelastpickle.com> wrote:
> >> Can I just copy data files for the required keyspaces, create schema
> >> manually and run repair?
> > If you have something like RF 3 and 3 nodes then yes, you can copy the data
> > from one node in the source cluster to all nodes in the dest cluster and use
> > cleanup to remove the unneeded data. Because each node in the source cluster
> > has a full copy of the data.
> >
> > If that's not the case you cannot copy the data files, even if they have the
> > same number of nodes, because the nodes in the dest cluster will have
> > different tokens. AFAIK you need to export the full data set from the source
> > DC and then import it into the dest system.
> >
> > The Bulk Load utility may be of help
> > http://www.datastax.com/docs/1.2/references/bulkloader . You could copy the
> > SSTables from every node in the source system and bulk load them into the
> > dest system. That process will ensure rows are sent to nodes that are
> > replicas.
> >
> > Cheers
> >
> > -----------------
> > Aaron Morton
> > Freelance Cassandra Consultant
> > New Zealand
> >
> > @aaronmorton
> > http://www.thelastpickle.com
> >
> > On 9/07/2013, at 12:45 PM, Baskar Duraikannu
> > <ba...@gmail.com> wrote:
> >
> >> We have two clusters used by two different groups with vnodes enabled. Now
> >> there is a need to move some of the keyspaces from cluster 1 to cluster 2.
> >>
> >>
> >> Can I just copy data files for the required keyspaces, create schema
> >> manually and run repair?
> >>
> >> Anything else required? Please help.
> >> --
> >> Thanks,
> >> Baskar Duraikannu
> >
> >
>
>
Re: Node tokens / data move
Posted by Baskar Duraikannu <ba...@gmail.com>.
I copied the sstables and then ran a repair. It worked. Looks like export
and import may have been much faster given that we had very little data.
Thanks everyone.
On Tue, Jul 9, 2013 at 1:34 PM, sankalp kohli <ko...@gmail.com>wrote:
> Hi Aaron,
> Can he not specify all 256 tokens in the YAML of the new
> cluster and then copy sstables?
> I know it is a bit ugly but should work.
>
> Sankalp
>
>
> On Tue, Jul 9, 2013 at 3:19 AM, Baskar Duraikannu <
> baskar.duraikannu.db@gmail.com> wrote:
>
>> Thanks Aaron
>>
>> On 7/9/13, aaron morton <aa...@thelastpickle.com> wrote:
>> >> Can I just copy data files for the required keyspaces, create schema
>> >> manually and run repair?
>> > If you have something like RF 3 and 3 nodes then yes, you can copy the
>> data
>> > from one node in the source cluster to all nodes in the dest cluster
>> and use
>> > cleanup to remove the unneeded data. Because each node in the source
>> cluster
>> > has a full copy of the data.
>> >
>> > If that's not the case you cannot copy the data files, even if they
>> have the
>> > same number of nodes, because the nodes in the dest cluster will have
>> > different tokens. AFAIK you need to export the full data set from the
>> source
>> > DC and then import it into the dest system.
>> >
>> > The Bulk Load utility may be of help
>> > http://www.datastax.com/docs/1.2/references/bulkloader . You could
>> copy the
>> > SSTables from every node in the source system and bulk load them into
>> the
>> > dest system. That process will ensure rows are sent to nodes that are
>> > replicas.
>> >
>> > Cheers
>> >
>> > -----------------
>> > Aaron Morton
>> > Freelance Cassandra Consultant
>> > New Zealand
>> >
>> > @aaronmorton
>> > http://www.thelastpickle.com
>> >
>> > On 9/07/2013, at 12:45 PM, Baskar Duraikannu
>> > <ba...@gmail.com> wrote:
>> >
>> >> We have two clusters used by two different groups with vnodes enabled.
>> Now
>> >> there is a need to move some of the keyspaces from cluster 1 to
>> cluster 2.
>> >>
>> >>
>> >> Can I just copy data files for the required keyspaces, create schema
>> >> manually and run repair?
>> >>
>> >> Anything else required? Please help.
>> >> --
>> >> Thanks,
>> >> Baskar Duraikannu
>> >
>> >
>>
>
>
Re: Node tokens / data move
Posted by sankalp kohli <ko...@gmail.com>.
Hi Aaron,
Can he not specify all 256 tokens in the YAML of the new
cluster and then copy sstables?
I know it is a bit ugly but should work.
Sankalp
On Tue, Jul 9, 2013 at 3:19 AM, Baskar Duraikannu <
baskar.duraikannu.db@gmail.com> wrote:
> Thanks Aaron
>
> On 7/9/13, aaron morton <aa...@thelastpickle.com> wrote:
> >> Can I just copy data files for the required keyspaces, create schema
> >> manually and run repair?
> > If you have something like RF 3 and 3 nodes then yes, you can copy the
> data
> > from one node in the source cluster to all nodes in the dest cluster and
> use
> > cleanup to remove the unneeded data. Because each node in the source
> cluster
> > has a full copy of the data.
> >
> > If that's not the case you cannot copy the data files, even if they have
> the
> > same number of nodes, because the nodes in the dest cluster will have
> > different tokens. AFAIK you need to export the full data set from the
> source
> > DC and then import it into the dest system.
> >
> > The Bulk Load utility may be of help
> > http://www.datastax.com/docs/1.2/references/bulkloader . You could copy
> the
> > SSTables from every node in the source system and bulk load them into the
> > dest system. That process will ensure rows are sent to nodes that are
> > replicas.
> >
> > Cheers
> >
> > -----------------
> > Aaron Morton
> > Freelance Cassandra Consultant
> > New Zealand
> >
> > @aaronmorton
> > http://www.thelastpickle.com
> >
> > On 9/07/2013, at 12:45 PM, Baskar Duraikannu
> > <ba...@gmail.com> wrote:
> >
> >> We have two clusters used by two different groups with vnodes enabled.
> Now
> >> there is a need to move some of the keyspaces from cluster 1 to cluster
> 2.
> >>
> >>
> >> Can I just copy data files for the required keyspaces, create schema
> >> manually and run repair?
> >>
> >> Anything else required? Please help.
> >> --
> >> Thanks,
> >> Baskar Duraikannu
> >
> >
>
Re: Node tokens / data move
Posted by Baskar Duraikannu <ba...@gmail.com>.
Thanks Aaron
On 7/9/13, aaron morton <aa...@thelastpickle.com> wrote:
>> Can I just copy data files for the required keyspaces, create schema
>> manually and run repair?
> If you have something like RF 3 and 3 nodes then yes, you can copy the data
> from one node in the source cluster to all nodes in the dest cluster and use
> cleanup to remove the unneeded data. Because each node in the source cluster
> has a full copy of the data.
>
> If that's not the case you cannot copy the data files, even if they have the
> same number of nodes, because the nodes in the dest cluster will have
> different tokens. AFAIK you need to export the full data set from the source
> DC and then import it into the dest system.
>
> The Bulk Load utility may be of help
> http://www.datastax.com/docs/1.2/references/bulkloader . You could copy the
> SSTables from every node in the source system and bulk load them into the
> dest system. That process will ensure rows are sent to nodes that are
> replicas.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 9/07/2013, at 12:45 PM, Baskar Duraikannu
> <ba...@gmail.com> wrote:
>
>> We have two clusters used by two different groups with vnodes enabled. Now
>> there is a need to move some of the keyspaces from cluster 1 to cluster 2.
>>
>>
>> Can I just copy data files for the required keyspaces, create schema
>> manually and run repair?
>>
>> Anything else required? Please help.
>> --
>> Thanks,
>> Baskar Duraikannu
>
>
Re: Node tokens / data move
Posted by aaron morton <aa...@thelastpickle.com>.
> Can I just copy data files for the required keyspaces, create schema manually and run repair?
If you have something like RF 3 and 3 nodes then yes, you can copy the data from one node in the source cluster to all nodes in the dest cluster and use cleanup to remove the unneeded data. Because each node in the source cluster has a full copy of the data.
If that's not the case you cannot copy the data files, even if they have the same number of nodes, because the nodes in the dest cluster will have different tokens. AFAIK you need to export the full data set from the source DC and then import it into the dest system.
The Bulk Load utility may be of help http://www.datastax.com/docs/1.2/references/bulkloader . You could copy the SSTables from every node in the source system and bulk load them into the dest system. That process will ensure rows are sent to nodes that are replicas.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 9/07/2013, at 12:45 PM, Baskar Duraikannu <ba...@gmail.com> wrote:
> We have two clusters used by two different groups with vnodes enabled. Now there is a need to move some of the keyspaces from cluster 1 to cluster 2.
>
> Can I just copy data files for the required keyspaces, create schema manually and run repair?
>
> Anything else required? Please help.
> --
> Thanks,
> Baskar Duraikannu