You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Gurpreet Singh <gu...@gmail.com> on 2010/09/17 00:19:51 UTC
questions on cassandra (repair and multi-datacenter)
Hi,
I have a few questions and was looking for an answer.
I have a cluster of 7 Cassandra 0.6.5 nodes in my test setup. RF=2. Original
data size is about 100 gigs, with RF=2, i see the total load on the cluster
is about 200 gigs, all good.
1. I was looking to increase the RF to 3. This process entails changing the
config and calling repair on the keyspace one at a time, right?
So, I started with one node at a time, changed the config file on the first
node for the keyspace, restarted the node. And then called a nodetool repair
on the node. These same steps i followed for every node after that, as I
read somewhere that the repair should be invoked one node at a time.
(a) What is the best way to ascertain if the repair is completed on a node?
(b) After the repair was finished, I was expecting the total data load to be
300 gigs. However, calling the ring command, shows the total load to be 370
gigs. I double checked and config on all machines says RF=3. I am calling a
cleanup on each node right now. Is the cleanup required after calling a
repair? Am i missing something?
2. This question is regarding multi-datacenter support. I plan to have a
cluster of 6 machines across 2 datacenters, with the machines from the
datacenters alternating on the ring. RF=3 is the plan. I already have a test
setup as described above, which has most of the data, but its still
configured on the default RackUnAware strategy. I was hoping to find the
right steps to move it to RackAware strategy with the
PropertyFileEndpointSnitch that I read somewhere (not sure if thats
supported in 0.6.5, but CustomEndPointSnitch is the same, right?), all this
without having to repopulate any data again.
Currently there is only 1 datacenter, but I was stil planning to set the
cluster up as it would be in multi-datacenter support, and run it like that
in the one datacenter, and when the second datacenter comes up, just copy
all the files across to the new nodes in the second datacenter, and bring
the whole cluster up. Will this work ? I have tried copying files to a new
node, shutting down all nodes, and bringing back everything up, and it
recognized the new ips.
Thanks
Gurpreet
Re: questions on cassandra (repair and multi-datacenter)
Posted by Gurpreet Singh <gu...@gmail.com>.
Hi Benjamin,
I reverted back to the old RF of 2, by restarting all nodes with RF 2, and
then running cleanup. It came down to 2.
This time, i now changed the RF to 3 for all machines and restarted all the
nodes.
I started running repair one by one on all machines, tracking through
jconsole that compaction (readonly) was happenning, and moved over to the
next node only when there was no compaction going on, and nothing in the AES
stage.
The process is finished on all machines, and took a long time i have to say.
However my ring shows almost 8 times load. Here it is.
Original data size was about 100 gigs, with RF 2 it was about 200 gigs. But
here its almost 800 gigs. What could i be doing wrong?
Address Status Load Range
Ring
128045052799293308972222897669231312075
ip1 Up 100.09 GB 418948754358022090024968091731975038
|<--|
ip2 Up 90.83 GB 11057191649494821574112431245768166381 |
^
ip3 Up 105.9 GB 21705674247570520134925875025273670789 v
|
ip4 Up 122.7 GB 42980788726886850121690234508345696103 |
^
ip5 Up 106.24 GB 85510669386552904928042350150568641153 v
|
ip6 Up 194.27 GB 106767287274351479790232508363491106683 |
^
ip7 Up 87.3 GB 128045052799293308972222897669231312075
|-->|
/G
On Thu, Sep 16, 2010 at 11:56 PM, Gurpreet Singh
<gu...@gmail.com>wrote:
> Thanks Benjamin. I realised that, i have reverted using cleanup, got it
> back to old state and testing the scenario exactly the way you put it.
>
>
> On Thu, Sep 16, 2010 at 10:56 PM, Benjamin Black <b...@b3k.us> wrote:
>
>> On Thu, Sep 16, 2010 at 3:19 PM, Gurpreet Singh
>> <gu...@gmail.com> wrote:
>> > 1. I was looking to increase the RF to 3. This process entails changing
>> the
>> > config and calling repair on the keyspace one at a time, right?
>> > So, I started with one node at a time, changed the config file on the
>> first
>> > node for the keyspace, restarted the node. And then called a nodetool
>> repair
>> > on the node.
>>
>> You need to change the RF on _all_ nodes in the cluster _before_
>> running repair on _any_ of them. If nodes disagree on which nodes
>> should have replicas for keys, repair will not work correctly.
>> Different RF for the same keyspace creates that disagreement.
>>
>>
>> b
>>
>
>
Re: questions on cassandra (repair and multi-datacenter)
Posted by Gurpreet Singh <gu...@gmail.com>.
Thanks Benjamin. I realised that, i have reverted using cleanup, got it back
to old state and testing the scenario exactly the way you put it.
On Thu, Sep 16, 2010 at 10:56 PM, Benjamin Black <b...@b3k.us> wrote:
> On Thu, Sep 16, 2010 at 3:19 PM, Gurpreet Singh
> <gu...@gmail.com> wrote:
> > 1. I was looking to increase the RF to 3. This process entails changing
> the
> > config and calling repair on the keyspace one at a time, right?
> > So, I started with one node at a time, changed the config file on the
> first
> > node for the keyspace, restarted the node. And then called a nodetool
> repair
> > on the node.
>
> You need to change the RF on _all_ nodes in the cluster _before_
> running repair on _any_ of them. If nodes disagree on which nodes
> should have replicas for keys, repair will not work correctly.
> Different RF for the same keyspace creates that disagreement.
>
>
> b
>
Re: questions on cassandra (repair and multi-datacenter)
Posted by Benjamin Black <b...@b3k.us>.
On Thu, Sep 16, 2010 at 3:19 PM, Gurpreet Singh
<gu...@gmail.com> wrote:
> 1. I was looking to increase the RF to 3. This process entails changing the
> config and calling repair on the keyspace one at a time, right?
> So, I started with one node at a time, changed the config file on the first
> node for the keyspace, restarted the node. And then called a nodetool repair
> on the node.
You need to change the RF on _all_ nodes in the cluster _before_
running repair on _any_ of them. If nodes disagree on which nodes
should have replicas for keys, repair will not work correctly.
Different RF for the same keyspace creates that disagreement.
b