You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Todd Nine <to...@gmail.com> on 2013/01/08 17:41:45 UTC
Script to load sstables from v1.0.x to v 1.1.x
Hi all,
I have recently been trying to restore backups from a v1.0.x cluster we
have into a 1.1.7 cluster. This has not been as trivial as I expected, and
I've had a lot of help from the IRC channel in tackling this problem. As a
way of saying thanks, I'd like to contribute the updated ruby script I was
originally given for accomplishing this task. Here it is.
https://gist.github.com/1c161edab88a4e4aea06
It takes a keyspace directory as the input, then creates symlinks in the
output directory with the 1.1 structure pointing to the 1.0 sstables. If
you've specified a host, it will then invoke the sstableloader for each of
the Keyspaces and CFs it discovers in the output directory. I hope this is
helpful to someone else. I'll keep the gist updated as I update the script.
Todd
Re: Script to load sstables from v1.0.x to v 1.1.x
Posted by Rob Coli <rc...@palominodb.com>.
On Tue, Jan 8, 2013 at 11:56 AM, Todd Nine <to...@gmail.com> wrote:
> Our current production
> cluster is still on 1.0.x, so we can either fire up a 1.0.x cluster, then
> upgrade every node to accomplish this, or just use the script.
No 1.0 cluster is required to restore 1.0 directory structure to a 1.1
cluster and have the tables be migrated by Cassandra. The 1.1 node
should look at the 1.0 directory structure you just restored and
migrate it automagically.
> We also have
> a different number of nodes in stage vs production, so we'd still need to
> run a repair if we did a straight sstable copy.
This is a compelling reason to bulk load. My commentary merely points
out that if you *aren't* changing cluster size/topology, Cassandra 1.1
should be migrating the sstables for you. :)
=Rob
--
=Robert Coli
AIM>ALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb
Re: Script to load sstables from v1.0.x to v 1.1.x
Posted by Todd Nine <to...@gmail.com>.
Our use case is for testing migrations in our data, as well as stress testing outside our production environment. To do this, we load our backups into a fresh cluster, then perform our testing. Our current production cluster is still on 1.0.x, so we can either fire up a 1.0.x cluster, then upgrade every node to accomplish this, or just use the script. We also have a different number of nodes in stage vs production, so we'd still need to run a repair if we did a straight sstable copy. The script is a lot faster and easier for us than going through the upgrade process, then running repair to ensure the data is distributed correctly in the ring.
--
Todd Nine
On Tuesday, January 8, 2013 at 12:32 PM, Michael Kjellman wrote:
> I thought this was to load between separate clusters not to upgrade within the same cluster. No?
>
> On Jan 8, 2013, at 11:29 AM, "Rob Coli" <rcoli@palominodb.com (mailto:rcoli@palominodb.com)> wrote:
>
> > On Tue, Jan 8, 2013 at 8:41 AM, Todd Nine <todd.nine@gmail.com (mailto:todd.nine@gmail.com)> wrote:
> > > I have recently been trying to restore backups from a v1.0.x cluster we
> > > have into a 1.1.7 cluster. This has not been as trivial as I expected, and
> > > I've had a lot of help from the IRC channel in tackling this problem. As a
> > > way of saying thanks, I'd like to contribute the updated ruby script I was
> > > originally given for accomplishing this task. Here it is.
> > >
> >
> >
> > While I laud your contribution, I am still not fully understanding why
> > this is not working "automagically", as it should :
> >
> > http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-1-flexible-data-file-placement
> > "
> > What about upgrading?
> >
> > Do you need to manually move all pre-1.1 data files to the new
> > directory structure before upgrading to 1.1? No. Immediately after
> > Cassandra 1.1 starts, it checks to see whether it has old directory
> > structure and migrates all data files (including backups and
> > snapshots) to the new directory structure if needed. So, just upgrade
> > as you always do (don’t forget to read NEWS.txt first), and you will
> > get more control over data files for free.
> > "
> >
> > Is it possible that, for example, the installation of the debian
> > package results in your 1.1.x node starting up before you intend it
> > to.. and then when you start it again with the 1.0 paths, it doesn't
> > try to change the paths?
> >
> > " * To check if sstables needs migration, we look at the System
> > directory. If it contains a directory for the status cf, we'll attempt
> > a sstable migrating. "
> >
> > This quote from Directories.java (thx driftx!) suggests that any
> > starting of a 1.1 node, which would result in a "Status" columnfamily
> > being created, would make sstablesNeedsMigration return false.
> >
> > If this is your case due to the use of the debian package or similar
> > which auto-starts, your input is welcomed at :
> >
> > https://issues.apache.org/jira/browse/CASSANDRA-2356
> >
> > =Rob
> >
> > --
> > =Robert Coli
> > AIM>ALK - rcoli@palominodb.com (mailto:rcoli@palominodb.com)
> > YAHOO - rcoli.palominob
> > SKYPE - rcoli_palominodb
> >
>
>
> Southfield Public School students safely access the tech tools they need on and off campus with the Barracuda Web Filter.
>
> Quick installation and easy to use- try the Barracuda Web Filter free for 30 days: http://on.fb.me/Vj6JBd
Re: Script to load sstables from v1.0.x to v 1.1.x
Posted by Michael Kjellman <mk...@barracuda.com>.
I thought this was to load between separate clusters not to upgrade within the same cluster. No?
On Jan 8, 2013, at 11:29 AM, "Rob Coli" <rc...@palominodb.com> wrote:
> On Tue, Jan 8, 2013 at 8:41 AM, Todd Nine <to...@gmail.com> wrote:
>> I have recently been trying to restore backups from a v1.0.x cluster we
>> have into a 1.1.7 cluster. This has not been as trivial as I expected, and
>> I've had a lot of help from the IRC channel in tackling this problem. As a
>> way of saying thanks, I'd like to contribute the updated ruby script I was
>> originally given for accomplishing this task. Here it is.
>
> While I laud your contribution, I am still not fully understanding why
> this is not working "automagically", as it should :
>
> http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-1-flexible-data-file-placement
> "
> What about upgrading?
>
> Do you need to manually move all pre-1.1 data files to the new
> directory structure before upgrading to 1.1? No. Immediately after
> Cassandra 1.1 starts, it checks to see whether it has old directory
> structure and migrates all data files (including backups and
> snapshots) to the new directory structure if needed. So, just upgrade
> as you always do (don’t forget to read NEWS.txt first), and you will
> get more control over data files for free.
> "
>
> Is it possible that, for example, the installation of the debian
> package results in your 1.1.x node starting up before you intend it
> to.. and then when you start it again with the 1.0 paths, it doesn't
> try to change the paths?
>
> " * To check if sstables needs migration, we look at the System
> directory. If it contains a directory for the status cf, we'll attempt
> a sstable migrating. "
>
> This quote from Directories.java (thx driftx!) suggests that any
> starting of a 1.1 node, which would result in a "Status" columnfamily
> being created, would make sstablesNeedsMigration return false.
>
> If this is your case due to the use of the debian package or similar
> which auto-starts, your input is welcomed at :
>
> https://issues.apache.org/jira/browse/CASSANDRA-2356
>
> =Rob
>
> --
> =Robert Coli
> AIM>ALK - rcoli@palominodb.com
> YAHOO - rcoli.palominob
> SKYPE - rcoli_palominodb
Southfield Public School students safely access the tech tools they need on and off campus with the Barracuda Web Filter.
Quick installation and easy to use- try the Barracuda Web Filter free for 30 days: http://on.fb.me/Vj6JBd
Re: Script to load sstables from v1.0.x to v 1.1.x
Posted by Rob Coli <rc...@palominodb.com>.
On Tue, Jan 8, 2013 at 8:41 AM, Todd Nine <to...@gmail.com> wrote:
> I have recently been trying to restore backups from a v1.0.x cluster we
> have into a 1.1.7 cluster. This has not been as trivial as I expected, and
> I've had a lot of help from the IRC channel in tackling this problem. As a
> way of saying thanks, I'd like to contribute the updated ruby script I was
> originally given for accomplishing this task. Here it is.
While I laud your contribution, I am still not fully understanding why
this is not working "automagically", as it should :
http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-1-flexible-data-file-placement
"
What about upgrading?
Do you need to manually move all pre-1.1 data files to the new
directory structure before upgrading to 1.1? No. Immediately after
Cassandra 1.1 starts, it checks to see whether it has old directory
structure and migrates all data files (including backups and
snapshots) to the new directory structure if needed. So, just upgrade
as you always do (don’t forget to read NEWS.txt first), and you will
get more control over data files for free.
"
Is it possible that, for example, the installation of the debian
package results in your 1.1.x node starting up before you intend it
to.. and then when you start it again with the 1.0 paths, it doesn't
try to change the paths?
" * To check if sstables needs migration, we look at the System
directory. If it contains a directory for the status cf, we'll attempt
a sstable migrating. "
This quote from Directories.java (thx driftx!) suggests that any
starting of a 1.1 node, which would result in a "Status" columnfamily
being created, would make sstablesNeedsMigration return false.
If this is your case due to the use of the debian package or similar
which auto-starts, your input is welcomed at :
https://issues.apache.org/jira/browse/CASSANDRA-2356
=Rob
--
=Robert Coli
AIM>ALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb