Re: -pr vs. no -pr

["Takenori Sato(Cloudian)" <ts...@cloudian.com>]

Hi,

Please note that I confirmed on v1.0.7.

 > I mean a repair involves all three nodes and pushes and pulls data, 
right?

Yes, but that's how -pr works. A repair without -pr does more.

For example, suppose you have a ring with RF=3 like this.

A - B - C - D - E - F

Then, a repair on A without -pr does for 3 ranges as follows:
[A, B, C]
[E, F, A]
[F, A, B]

Among them, the first one, [A, B, C] is the primary range of A.

So, with -pr, a repair runs only for:
[A, B, C]

 > I could run nodetool repair on just 2 nodes(RF=3) instead of using 
nodetool repair –pr???

Yes.

You need to run two repairs on A and D.

 > What is the advantage of –pr then?

Whenever you want to minimize rapair impacts.

For example, suppose you got one node down for a while, and bring it 
back to the cluster.

You need to run rapair without affecting the entire cluster. Then, -pr 
is the option.

Thanks,
Takenori

(2013/03/01 7:39), Hiller, Dean wrote:
> Isn't it true if I have 6 nodes, I could run nodetool repair on just 2 nodes(RF=3) instead of using nodetool repair –pr???
>
> What is the advantage of –pr then?
>
> I mean a repair involves all three nodes and pushes and pulls data, right?
>
> Thanks,
> Dean

Re: -pr vs. no -pr

[Michael Theroux <mt...@yahoo.com>]

The way I've always thought about it is that -pr will make sure the information that specific node originates is consistent with its replicas.

So, we know that a node is responsible for a specific token range, and the next nodes in the ring will hold its replicas.  The -pr will make sure that a specific node's information is consistent to its replicas, but will not make sure a specific node has all the replicated information it can get from nodes previous to itself in the ring.

Without the -pr option, not only will the current node make sure its information and its replica's information is consistent, but it will also make sure that all the information that it is a replica for, is consistent.  

If you run regular repairs on all the nodes in your cluster, then -pr is sufficient.  Every node will run repair, and make sure its information is consistent with its replicas, eventually creating a fully consistent cluster.  This is a quicker process, and will have less impact on your operations by essentially spreading out the pain.  

For instance, we run a 12 node cluster.  We run "nodetool repair -pr" on nodes that are opposite to each other, 4 nodes a day (2 nodes in the morning, 2 nodes in the evening).  With a grace period of 10 days, this allows us to run repairs twice a week on a specific node, and to occasionally skip repairs on specific nodes once a week.  

In this case, without -pr, a lot of extra work would be done.  In fact, with an RF of 3 (in our case), the time per repair would increase many fold.

Another way to thing about it... although likely not 100% technically correct..

A repair -pr will cause a push of a node's information to its replicas.  Without the -pr, it will cause a push, and it will cause nodes it is a replica for to push their information as well.

-Mike

On Feb 28, 2013, at 9:39 PM, Hiller, Dean wrote:

> Isn't there more to it than that.  You really have nodes responsible for
> token ranges like so(using describe ring)
> 
> What we see is this from our describe ringŠ(1 to 6 are token ranges while
> A to F are servers)Š.
> A - 1, 2, 3
> B - 2, 3, 4
> C - 3, 4, 5
> D - 4, 5, 6
> E - 5, 6, 1
> F - 6, 1, 2
> 
> With -pr, only token range 1 is repaired I think, right?  2 and 3 are only
> repaired without the -pr option?  This means if I have a node that I just
> joined the cluster, I should "not" be using -pr as 2 and 3 on node A will
> not be up to date.  Using -pr is nice if I am going to repair every single
> node and is nice for the cron job that has to happen before
> gc_grace_seconds.  Am I wrong here?  Ie. -pr is really only good for use
> in the cron job as it would miss 2 and 3 above.  I could run the cron on
> just two servers but then my nodes are different which can be a hassle.
> 
> Please verify that is what you believe is what happens as well?
> 
> Thanks,
> Dean
> 
> On 2/28/13 5:58 PM, "Takenori Sato(Cloudian)" <ts...@cloudian.com> wrote:
> 
>> Hi,
>> 
>> Please note that I confirmed on v1.0.7.
>> 
>>> I mean a repair involves all three nodes and pushes and pulls data,
>> right?
>> 
>> Yes, but that's how -pr works. A repair without -pr does more.
>> 
>> For example, suppose you have a ring with RF=3 like this.
>> 
>> A - B - C - D - E - F
>> 
>> Then, a repair on A without -pr does for 3 ranges as follows:
>> [A, B, C]
>> [E, F, A]
>> [F, A, B]
>> 
>> Among them, the first one, [A, B, C] is the primary range of A.
>> 
>> So, with -pr, a repair runs only for:
>> [A, B, C]
>> 
>>> I could run nodetool repair on just 2 nodes(RF=3) instead of using
>> nodetool repair ­pr???
>> 
>> Yes.
>> 
>> You need to run two repairs on A and D.
>> 
>>> What is the advantage of ­pr then?
>> 
>> Whenever you want to minimize rapair impacts.
>> 
>> For example, suppose you got one node down for a while, and bring it
>> back to the cluster.
>> 
>> You need to run rapair without affecting the entire cluster. Then, -pr
>> is the option.
>> 
>> Thanks,
>> Takenori
>> 
>> (2013/03/01 7:39), Hiller, Dean wrote:
>>> Isn't it true if I have 6 nodes, I could run nodetool repair on just 2
>>> nodes(RF=3) instead of using nodetool repair ­pr???
>>> 
>>> What is the advantage of ­pr then?
>>> 
>>> I mean a repair involves all three nodes and pushes and pulls data,
>>> right?
>>> 
>>> Thanks,
>>> Dean
>> 
> 

Re: -pr vs. no -pr

["Hiller, Dean" <De...@nrel.gov>]

Isn't there more to it than that.  You really have nodes responsible for
token ranges like so(using describe ring)

What we see is this from our describe ringŠ(1 to 6 are token ranges while
A to F are servers)Š.
A - 1, 2, 3
B - 2, 3, 4
C - 3, 4, 5
D - 4, 5, 6
E - 5, 6, 1
F - 6, 1, 2

With -pr, only token range 1 is repaired I think, right?  2 and 3 are only
repaired without the -pr option?  This means if I have a node that I just
joined the cluster, I should "not" be using -pr as 2 and 3 on node A will
not be up to date.  Using -pr is nice if I am going to repair every single
node and is nice for the cron job that has to happen before
gc_grace_seconds.  Am I wrong here?  Ie. -pr is really only good for use
in the cron job as it would miss 2 and 3 above.  I could run the cron on
just two servers but then my nodes are different which can be a hassle.

Please verify that is what you believe is what happens as well?

Thanks,
Dean

On 2/28/13 5:58 PM, "Takenori Sato(Cloudian)" <ts...@cloudian.com> wrote:

>Hi,
>
>Please note that I confirmed on v1.0.7.
>
> > I mean a repair involves all three nodes and pushes and pulls data,
>right?
>
>Yes, but that's how -pr works. A repair without -pr does more.
>
>For example, suppose you have a ring with RF=3 like this.
>
>A - B - C - D - E - F
>
>Then, a repair on A without -pr does for 3 ranges as follows:
>[A, B, C]
>[E, F, A]
>[F, A, B]
>
>Among them, the first one, [A, B, C] is the primary range of A.
>
>So, with -pr, a repair runs only for:
>[A, B, C]
>
> > I could run nodetool repair on just 2 nodes(RF=3) instead of using
>nodetool repair ­pr???
>
>Yes.
>
>You need to run two repairs on A and D.
>
> > What is the advantage of ­pr then?
>
>Whenever you want to minimize rapair impacts.
>
>For example, suppose you got one node down for a while, and bring it
>back to the cluster.
>
>You need to run rapair without affecting the entire cluster. Then, -pr
>is the option.
>
>Thanks,
>Takenori
>
>(2013/03/01 7:39), Hiller, Dean wrote:
>> Isn't it true if I have 6 nodes, I could run nodetool repair on just 2
>>nodes(RF=3) instead of using nodetool repair ­pr???
>>
>> What is the advantage of ­pr then?
>>
>> I mean a repair involves all three nodes and pushes and pulls data,
>>right?
>>
>> Thanks,
>> Dean
>