You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by horschi <ho...@gmail.com> on 2013/12/23 14:15:18 UTC
Offline migration: Random->Murmur
Hi list,
has anyone ever tried to migrate a cluster from Random to Murmur?
We would like to do so, to have a more standardized setup. I wrote a small
(yet untested) utility, which should be able to read SSTable files from
disk and write them into a cassandra cluster using Hector. This migration
would be offline of course and would only work for smaller clusters.
Any thoughts on the topic?
kind regards,
Christian
PS: The reason for doing so are not "performance". It is to simplify
operational stuff for the years to come. :-)
Re: Offline migration: Random->Murmur
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Dec 30, 2013 at 6:15 PM, Aaron Morton <aa...@thelastpickle.com>wrote:
> I wrote a small (yet untested) utility, which should be able to read
> SSTable files from disk and write them into a cassandra cluster using
> Hector.
>
> Consider using the SSTableSimpleUnsortedWriter (see
> http://www.datastax.com/dev/blog/bulk-loading) to create the SSTables you
> can then bulk load them into the destination system.This will be much
> faster.
>
More details here :
http://www.palominodb.com/blog/2012/09/25/bulk-loading-options-cassandra
=Rob
Re: Offline migration: Random->Murmur
Posted by Aaron Morton <aa...@thelastpickle.com>.
> I wrote a small (yet untested) utility, which should be able to read SSTable files from disk and write them into a cassandra cluster using Hector.
Consider using the SSTableSimpleUnsortedWriter (see http://www.datastax.com/dev/blog/bulk-loading) to create the SSTables you can then bulk load them into the destination system.This will be much faster.
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 29/12/2013, at 6:26 am, Edward Capriolo <ed...@gmail.com> wrote:
> Internally we have a tool that does get range slice on the souce cluster and replicates to destination.
>
> Remeber that writes are itempotemt. Our tool can optionally only replicate data between two timestamps, allowing incremental transfers.
>
> So if you get your application writing new data to both clusters you can run a range scanning program to copy all the data.
>
> On Monday, December 23, 2013, horschi <ho...@gmail.com> wrote:
> > Interesting you even dare to do a live migration :-)
> >
> > Do you do all Murmur-writes with the timestamp from the "Random"-data? So that all migrated data is written with timestamps from the past.
> >
> >
> >
> > On Mon, Dec 23, 2013 at 3:59 PM, Rahul Menon <ra...@apigee.com> wrote:
> >>
> >> Christian,
> >>
> >> I have been planning to migrate my cluster from random to murmur3 in a similar manner. I intend to use pycassa to read and then write to the newer cluster. My only concern would be ensuring the consistency of already migrated data as the cluster ( with random ) would be constantly serving the production traffic. I was able to do this on a non prod cluster, but production is a different game.
> >>
> >> I would also like to hear more about this, especially if someone was able to successfully do this.
> >>
> >> Thanks
> >> Rahul
> >>
> >>
> >> On Mon, Dec 23, 2013 at 6:45 PM, horschi <ho...@gmail.com> wrote:
> >>>
> >>> Hi list,
> >>>
> >>> has anyone ever tried to migrate a cluster from Random to Murmur?
> >>>
> >>> We would like to do so, to have a more standardized setup. I wrote a small (yet untested) utility, which should be able to read SSTable files from disk and write them into a cassandra cluster using Hector. This migration would be offline of course and would only work for smaller clusters.
> >>>
> >>> Any thoughts on the topic?
> >>>
> >>> kind regards,
> >>> Christian
> >>>
> >>> PS: The reason for doing so are not "performance". It is to simplify operational stuff for the years to come. :-)
> >>
> >
> >
>
> --
> Sorry this was sent from mobile. Will do less grammar and spell check than usual.
Re: Offline migration: Random->Murmur
Posted by Edward Capriolo <ed...@gmail.com>.
Internally we have a tool that does get range slice on the souce cluster
and replicates to destination.
Remeber that writes are itempotemt. Our tool can optionally only replicate
data between two timestamps, allowing incremental transfers.
So if you get your application writing new data to both clusters you can
run a range scanning program to copy all the data.
On Monday, December 23, 2013, horschi <ho...@gmail.com> wrote:
> Interesting you even dare to do a live migration :-)
>
> Do you do all Murmur-writes with the timestamp from the "Random"-data? So
that all migrated data is written with timestamps from the past.
>
>
>
> On Mon, Dec 23, 2013 at 3:59 PM, Rahul Menon <ra...@apigee.com> wrote:
>>
>> Christian,
>>
>> I have been planning to migrate my cluster from random to murmur3 in a
similar manner. I intend to use pycassa to read and then write to the newer
cluster. My only concern would be ensuring the consistency of already
migrated data as the cluster ( with random ) would be constantly serving
the production traffic. I was able to do this on a non prod cluster, but
production is a different game.
>>
>> I would also like to hear more about this, especially if someone was
able to successfully do this.
>>
>> Thanks
>> Rahul
>>
>>
>> On Mon, Dec 23, 2013 at 6:45 PM, horschi <ho...@gmail.com> wrote:
>>>
>>> Hi list,
>>>
>>> has anyone ever tried to migrate a cluster from Random to Murmur?
>>>
>>> We would like to do so, to have a more standardized setup. I wrote a
small (yet untested) utility, which should be able to read SSTable files
from disk and write them into a cassandra cluster using Hector. This
migration would be offline of course and would only work for smaller
clusters.
>>>
>>> Any thoughts on the topic?
>>>
>>> kind regards,
>>> Christian
>>>
>>> PS: The reason for doing so are not "performance". It is to simplify
operational stuff for the years to come. :-)
>>
>
>
--
Sorry this was sent from mobile. Will do less grammar and spell check than
usual.
Re: Offline migration: Random->Murmur
Posted by horschi <ho...@gmail.com>.
Interesting you even dare to do a live migration :-)
Do you do all Murmur-writes with the timestamp from the "Random"-data? So
that all migrated data is written with timestamps from the past.
On Mon, Dec 23, 2013 at 3:59 PM, Rahul Menon <ra...@apigee.com> wrote:
> Christian,
>
> I have been planning to migrate my cluster from random to murmur3 in a
> similar manner. I intend to use pycassa to read and then write to the newer
> cluster. My only concern would be ensuring the consistency of already
> migrated data as the cluster ( with random ) would be constantly serving
> the production traffic. I was able to do this on a non prod cluster, but
> production is a different game.
>
> I would also like to hear more about this, especially if someone was able
> to successfully do this.
>
> Thanks
> Rahul
>
>
> On Mon, Dec 23, 2013 at 6:45 PM, horschi <ho...@gmail.com> wrote:
>
>> Hi list,
>>
>> has anyone ever tried to migrate a cluster from Random to Murmur?
>>
>> We would like to do so, to have a more standardized setup. I wrote a
>> small (yet untested) utility, which should be able to read SSTable files
>> from disk and write them into a cassandra cluster using Hector. This
>> migration would be offline of course and would only work for smaller
>> clusters.
>>
>> Any thoughts on the topic?
>>
>> kind regards,
>> Christian
>>
>> PS: The reason for doing so are not "performance". It is to simplify
>> operational stuff for the years to come. :-)
>>
>
>
Re: Offline migration: Random->Murmur
Posted by Rahul Menon <ra...@apigee.com>.
Christian,
I have been planning to migrate my cluster from random to murmur3 in a
similar manner. I intend to use pycassa to read and then write to the newer
cluster. My only concern would be ensuring the consistency of already
migrated data as the cluster ( with random ) would be constantly serving
the production traffic. I was able to do this on a non prod cluster, but
production is a different game.
I would also like to hear more about this, especially if someone was able
to successfully do this.
Thanks
Rahul
On Mon, Dec 23, 2013 at 6:45 PM, horschi <ho...@gmail.com> wrote:
> Hi list,
>
> has anyone ever tried to migrate a cluster from Random to Murmur?
>
> We would like to do so, to have a more standardized setup. I wrote a small
> (yet untested) utility, which should be able to read SSTable files from
> disk and write them into a cassandra cluster using Hector. This migration
> would be offline of course and would only work for smaller clusters.
>
> Any thoughts on the topic?
>
> kind regards,
> Christian
>
> PS: The reason for doing so are not "performance". It is to simplify
> operational stuff for the years to come. :-)
>