You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Anthony Molinaro <an...@alumni.caltech.edu> on 2010/09/02 20:09:46 UTC
Data Center Move
Hi,
We're running cassandra 0.6.4, and need to do a data center move of
a cluster (from EC2 to our own data center). Because of the way the
networks are set up we can't actually connect these boxes directly, so
the original plan of add some nodes in the new colo, let them bootstrap
then decommission nodes in the old colo until the data is all transfered
will not work.
So I'm wondering if the following will work
1. take a snapshot on the source cluster
2. rsync all the files from the old machines to the new machines (we'd most
likely be reducing the total number of machines, so would do things like
take 4-5 machines worth of data and put it onto 1 machine)
3. bring up the new machines in the new colo
4. run cleanup on all new nodes?
5. run repair on all new nodes?
So will this work? If so, are steps 4 and 5 correct?
I realize we will miss any new data that happens between the snapshot
and turning on writes on the new cluster, but I think we might be able
to just tune compaction such that it doesn't happen, then just sync
the files that change while the data transfers happen?
Thanks,
-Anthony
--
------------------------------------------------------------------------
Anthony Molinaro <an...@alumni.caltech.edu>
Re: Data Center Move
Posted by Benjamin Black <b...@b3k.us>.
You will likely need to rename some of the files to avoid collisions
(they are only unique per node). Otherwise, yes, this can work.
On Thu, Sep 2, 2010 at 11:09 AM, Anthony Molinaro
<an...@alumni.caltech.edu> wrote:
> Hi,
>
> We're running cassandra 0.6.4, and need to do a data center move of
> a cluster (from EC2 to our own data center). Because of the way the
> networks are set up we can't actually connect these boxes directly, so
> the original plan of add some nodes in the new colo, let them bootstrap
> then decommission nodes in the old colo until the data is all transfered
> will not work.
>
> So I'm wondering if the following will work
>
> 1. take a snapshot on the source cluster
> 2. rsync all the files from the old machines to the new machines (we'd most
> likely be reducing the total number of machines, so would do things like
> take 4-5 machines worth of data and put it onto 1 machine)
> 3. bring up the new machines in the new colo
> 4. run cleanup on all new nodes?
> 5. run repair on all new nodes?
>
> So will this work? If so, are steps 4 and 5 correct?
>
> I realize we will miss any new data that happens between the snapshot
> and turning on writes on the new cluster, but I think we might be able
> to just tune compaction such that it doesn't happen, then just sync
> the files that change while the data transfers happen?
>
> Thanks,
>
> -Anthony
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro <an...@alumni.caltech.edu>
>
Re: Data Center Move
Posted by Anthony Molinaro <an...@alumni.caltech.edu>.
Hi,
Yes we saw that but felt that maybe moving files would be faster than
rerolling a patched version of the server, so we were wondering if we
could move files as described. If that fails to work for us, we may
try out this patch.
-Anthony
On Thu, Sep 02, 2010 at 01:28:15PM -0500, Peter Fales wrote:
> Anthony,
>
> I'm just getting my feet wet with Cassandra, so I'm far from an
> expert, but I'm curious whether you saw my posting a few days ago
> about using the EC2 "public" IP addreses with cassandra:
> http://www.mail-archive.com/user@cassandra.apache.org/msg05692.html
>
> *If* I understand the problem correctly, it seems like you could create
> some new EC2 nodes using this patched version of the code, then
> migrate your existing nodes to new EC2 nodes, but giving each new node a
> public IP. Once your entire EC2 cluster was up and running on the
> public addresses, you should be able to use those public addresses
> to migrate to some other site outside of EC2.
>
> Am I missing something obvious? (Quite possible, since I haven't actually
> tested this)
>
> On Thu, Sep 02, 2010 at 01:09:46PM -0500, Anthony Molinaro wrote:
> > Hi,
> >
> > We're running cassandra 0.6.4, and need to do a data center move of
> > a cluster (from EC2 to our own data center). Because of the way the
> > networks are set up we can't actually connect these boxes directly, so
> > the original plan of add some nodes in the new colo, let them bootstrap
> > then decommission nodes in the old colo until the data is all transfered
> > will not work.
> >
> > So I'm wondering if the following will work
> >
> > 1. take a snapshot on the source cluster
> > 2. rsync all the files from the old machines to the new machines (we'd most
> > likely be reducing the total number of machines, so would do things like
> > take 4-5 machines worth of data and put it onto 1 machine)
> > 3. bring up the new machines in the new colo
> > 4. run cleanup on all new nodes?
> > 5. run repair on all new nodes?
> >
> > So will this work? If so, are steps 4 and 5 correct?
> >
> > I realize we will miss any new data that happens between the snapshot
> > and turning on writes on the new cluster, but I think we might be able
> > to just tune compaction such that it doesn't happen, then just sync
> > the files that change while the data transfers happen?
> >
> > Thanks,
> >
> > -Anthony
> >
> > --
> > ------------------------------------------------------------------------
> > Anthony Molinaro <an...@alumni.caltech.edu>
>
> --
> Peter Fales
> Alcatel-Lucent
> Member of Technical Staff
> 1960 Lucent Lane
> Room: 9H-505
> Naperville, IL 60566-7033
> Email: Peter.Fales@alcatel-lucent.com
> Phone: 630 979 8031
--
------------------------------------------------------------------------
Anthony Molinaro <an...@alumni.caltech.edu>
Re: Data Center Move
Posted by Peter Fales <Pe...@alcatel-lucent.com>.
Anthony,
I'm just getting my feet wet with Cassandra, so I'm far from an
expert, but I'm curious whether you saw my posting a few days ago
about using the EC2 "public" IP addreses with cassandra:
http://www.mail-archive.com/user@cassandra.apache.org/msg05692.html
*If* I understand the problem correctly, it seems like you could create
some new EC2 nodes using this patched version of the code, then
migrate your existing nodes to new EC2 nodes, but giving each new node a
public IP. Once your entire EC2 cluster was up and running on the
public addresses, you should be able to use those public addresses
to migrate to some other site outside of EC2.
Am I missing something obvious? (Quite possible, since I haven't actually
tested this)
On Thu, Sep 02, 2010 at 01:09:46PM -0500, Anthony Molinaro wrote:
> Hi,
>
> We're running cassandra 0.6.4, and need to do a data center move of
> a cluster (from EC2 to our own data center). Because of the way the
> networks are set up we can't actually connect these boxes directly, so
> the original plan of add some nodes in the new colo, let them bootstrap
> then decommission nodes in the old colo until the data is all transfered
> will not work.
>
> So I'm wondering if the following will work
>
> 1. take a snapshot on the source cluster
> 2. rsync all the files from the old machines to the new machines (we'd most
> likely be reducing the total number of machines, so would do things like
> take 4-5 machines worth of data and put it onto 1 machine)
> 3. bring up the new machines in the new colo
> 4. run cleanup on all new nodes?
> 5. run repair on all new nodes?
>
> So will this work? If so, are steps 4 and 5 correct?
>
> I realize we will miss any new data that happens between the snapshot
> and turning on writes on the new cluster, but I think we might be able
> to just tune compaction such that it doesn't happen, then just sync
> the files that change while the data transfers happen?
>
> Thanks,
>
> -Anthony
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro <an...@alumni.caltech.edu>
--
Peter Fales
Alcatel-Lucent
Member of Technical Staff
1960 Lucent Lane
Room: 9H-505
Naperville, IL 60566-7033
Email: Peter.Fales@alcatel-lucent.com
Phone: 630 979 8031